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Abstract 

The  Air  Force  Class  A  aviation  mishap  rate  has  hovered  around  1.5  mishaps  per 
100,000  flight  hours  since  1985.  Recent  attention  on  Air  Force  accidents  has  caused  the 
leadership  to  seek  to  reduce  its  mishap  rate.  The  Army's  Class  A  aviation  mishap  rate 
declined  after  it  implemented  risk  management  (RM)  principles  in  1987.  This  reduction 
caught  the  attention  of  Air  Force  leadership  who  have  since  stated  that  the  application  of 
operational  risk  management  (ORM)  is  how  the  Air  Force  will  reduce,  even  eliminate, 
mishaps.  With  current  budget  constraints,  ORM  is  considered  to  be  the  most  cost- 
effective  way  the  Air  Force  can  reduce  its  mishap  rate. 

The  purpose  of  this  research  was  to  determine  whether  the  Air  Force  can  expect 
its  mishap  rate  to  significantly  decline  due  to  ORM  implementation.  This  determination 
is  based  on  the  relationship  between  the  Army's  implementation  of  RM  and  its  aviation 
mishap  rate.  The  analysis  of  the  Army’s  aviation  mishap  rates  and  available  causal  data 
was  performed  primarily  using  discontinuous  piecewise  linear  regression.  Results 
showed  that  the  effect  of  RM  was  not  reflected  in  the  Army's  mishap  rates.  As  a  result, 
the  Air  Force  should  not  expect  its  mishap  rate  to  significantly  decline  due  to  ORM 
implementation. 
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I.  Introduction 


Background 

Since  man  has  been  flying,  aircraft  accidents  have  taken  lives,  destroyed  property, 
and  damaged  or  destroyed  aircraft.  Between  1990  and  1996  alone  Department  of 
Defense  (DoD)  aviation  losses  totaled  $9.42  billion,  777  lost  lives,  and  741  destroyed 
aircraft  (Department  of  Defense,  1997:ES-2).  Flying  aircraft,  particularly  military 
aircraft  that  could  experience  the  hazard  of  combat,  is  an  inherently  risky  endeavor. 

Thus,  continuously  improving  military  flight  safety  is  a  high  priority  for  both  civilian  and 
military  leaders. 

In  June  1995,  Congressman  Ike  Skelton  of  Missouri  raised  concerns  over  Air 
Force  mishaps.  He  cited  accidents  such  as  the  B-52  crash  at  Fairchild  AFB,  the  T-38 
crash  in  Texas  that  hit  an  apartment  complex  and  the  infamous  F-15  friendly-fire  shoot- 
down  of  two  Army  Black  Hawk  helicopters  over  Iraq.  Skelton  proposed  that  training, 
flying  hours,  spare  part  resources,  and  high  operations  tempo  were  possible  factors  for 
these  mishaps  (Dorr,  1995:8,9). 

Whenever  an  aviation  mishap  occurs,  a  safety  investigation  ensues.  All  DoD 
flight  mishap  investigations  take  the  form  of  limited-use  safety  mishap  investigation 
reports.  An  investigation's  “SOLE  purpose  is  prevention  of  subsequent  DoD  mishaps” 
(Department  of  Defense,  1989).  However,  corrective  actions  to  prevent  mishaps  from 
reoccurring  can  be  costly  not  only  in  terms  of  dollars,  but  in  time  and  manpower. 
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Addressing  the  primary  causes  of  mishaps  and  taking  steps  to  prevent  mishaps  due  to 
these  causes  is  the  most  logical  course  of  action  to  improve  flight  safety.  Traditional 
mishap  prevention  has  centered  on  improving  design  of  aircraft  and  maintenance 
inspection  techniques.  In  an  increasingly  fiscally  constrained  operating  environment, 
these  methods  only  go  so  far  toward  mishap  reduction.  Other  avenues  must  be  sought 
and  applied  if  the  DoD  is  going  to  improve  its  safety  record. 

A  study  performed  by  the  Defense  Science  Board  Task  Force  on  Aviation  Safety 
reported  that  human  error  was  at  least  a  contributing  factor  in  over  70  percent  of  DoD 
Class  A  mishaps  (Department  of  Defense,  1997:31),  the  primary  measure  of  flight  safety. 
It  determined  that  the  practice  of  risk  management  (RM)  would  be  the  most  effective  and 
least  costly  method  of  mishap  reduction  (1997:3 1).  In  fact,  one  of  the  recommendations 
that  came  out  of  the  study  was  to  integrate  RM  practices  throughout  all  the  services 
(1997:ES-4).  In  a  letter  to  all  the  service  secretaries,  the  Deputy  Secretary  of  Defense 
suggested  a  goal  of  “zero  aviation  fatalities”  with  the  integration  of  RM  as  a  means  to 
that  end  (White,  1997). 

The  concept  of  RM  has  been  formally  applied  to  Army  operations  since  1987. 
Although  the  Army  did  see  a  reduction  in  its  Class  A  aviation  mishap  rate  after  the 
implementation  of  RM,  whether  or  not  the  reduction  was  due  to  RM  has  been 
inconclusive. 

Problem  Statement 

Although  the  DoD's  Class  A  mishap  rate  has  dropped  considerably  over  the 
history  of  aviation,  during  no  year  has  it  been  zero.  For  the  Air  Force,  there  has  been  no 
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significant  reduction  in  the  Class  A  mishap  rate  since  the  mid-1980’s.  It  has  hovered 
around  1.5  mishaps  per  100,000  flying  hours.  In  order  to  conserve  its  most  important 
resources,  people  and  war  machines,  the  Air  Force  is  looking  for  ways  to  reduce  its 
aviation  mishap  rate.  The  Army  aviation  mishap  rate  reduction  caught  the  attention  of 
Air  Force  leadership  who  have  since  stated  that  the  application  of  operational  risk 
management  (ORM)  is  how  the  Air  Force  will  reduce,  even  eliminate,  mishaps 
(Department  of  the  Air  Force,  1998b).  ORM  has  been  applied  on  an  experiential  and 
intuitive  basis  for  many  years  but  it  wasn’t  until  1996  that  the  Chief  of  Staff  approved 
implementation  of  ORM  Air  Force-wide.  With  current  budget  constraints  limiting 
investment  in  new  designs  and  inspection  tools,  ORM  is  viewed  to  be  the  most  cost- 
effective  way  the  Air  Force  can  reduce  its  mishap  rate. 

Research  Question 

Can  the  Air  Force  expect  the  mishap  rate  to  significantly  decline  after  ORM 
implementation?  The  answer  may  lie  in  looking  at  the  relationship  between  the  Army's 
implementation  of  RM  and  its  aviation  mishap  rate. 

Investigative  Questions 

1 .  What  are  the  major  factors  that  influence  military  aviation  safety? 

2.  How  does  the  Air  Force  implementation  of  ORM  compare  to  the  Army's 
implementation  of  RM? 

3.  Was  there  a  significant  difference  in  the  Army  aircraft  mishap  rates  after  the 
implementation  of  RM? 

a)  If  so,  was  the  difference  due  to  the  implementation  of  RM? 


3 


b)  If  there  was  a  significant  difference  in  the  mishap  rate,  how  much  of  an 


effect  was  the  implementation  of  RM? 


Scope 

This  research  deals  specifically  with  the  effect  of  risk  management  on  the  aviation 
mishap  rate  as  opposed  to  ground  and  off-duty  mishap  rates.  Flight-related  and  aircraft 
ground  mishaps  are  not  considered.  Flight  mishaps  as  defined  in  DODI  6055.7  are  the 
subject  at  hand.  Although  the  primary  focus  will  be  on  the  Army  Class  A  mishap  rate 
data.  Classes  B  and  C  mishap  data  will  also  be  analyzed  to  see  what  effect  risk 
management  may  have  had.  Although  the  Navy  and  Marine  Corps  have  active  ORM 
programs,  they,  like  the  Air  Force,  are  in  the  infancy  of  implementation.  Since  the  Army 
is  the  lead  service  concerning  risk  management,  this  research  will  predict  what  effect 
ORM  may  have  on  the  Air  Force  aviation  mishap  rate  based  on  the  effect  RM  has  had  on 
the  Army  aviation  mishap  rate.  An  assumption  for  this  research  is  that  after  RM 
implementation  into  the  Army,  aviators  incorporated  RM  practices  into  their  mission 
planning. 
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II.  Literature  Review 


Overview 

This  chapter  provides  a  background  of  aviation  safety  and  risk  management  as  it 
relates  to  the  Army  and  Air  Force.  Aviation  safety  factors  are  first  discussed,  describing 
the  typical  mishap  causes  and  mishap  prevention  methods.  Second,  a  history  of  the  risk 
management  concept  is  described.  Safety  and  risk  management  terms  are  then  defined  as 
they  pertain  to  the  Army  and  Air  Force.  Finally,  Army  and  Air  Force  risk  management 
implementation  is  examined. 

Aviation  Safety  Factors 

There  are  an  infinite  number  of  factors  that  can  affect  the  safety  of  any  given 
flight.  It  is  the  intention  here,  however,  to  identify  the  major  factors  recognized  by  the 
aviation  community  as  having  a  significant,  proven  impact  on  aviation  safety.  The 
discussion  will  take  two  related  paths.  First,  those  factors  typically  found  causal  to  a 
mishap  will  be  described.  Second,  the  prevailing  mishap  prevention  factors  will  be 
addressed. 

Mishap  Cause  Factors.  According  to  DODI  6055.7,  the  four  flight  mishap  cause 
classifications  are  human  factor,  material  failure,  environment,  and  other  (Department  of 
Defense,  1989).  These  factors  are  depicted  in  Figure  1.  The  instruction,  however,  does 
not  define  those  terms.  The  Army  uses  similar  terms  that  parallel  those  of  the  DoD 
instruction.  The  Army  replaces  the  human  factor  term  with  human  error,  material  failure 
with  materiel  factors,  and  does  not  have  an  “other”  category  (Department  of  the  Army, 
1999a).  The  Air  Force  uses  the  categories  of  people,  parts,  paper,  and  other  (Department 
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of  the  Air  Force,  1998c:64).  For  the  purposes  of  this  research,  the  DoD  terms  will  be 
used.  Another  term  that  has  received  some  attention  as  a  viable  mishap  cause  factor  the 
last  few  years  is  operational  tempo. 

Human  Factors 


Material  Failure 


rr 


Other 

Figure  1 .  Aviation  Mishap  Cause  Factors 
Human  Factors.  “Human  factors  in  aviation  safety  concerns  itself  with  the 
study  of  human  capabilities,  limitations,  and  behaviors  and  the  integration  of  that 
knowledge  into  a  system  design  with  the  goals  of  enhancing  safety  and  allowing  for  more 
efficient  operations”  (The  Southern  California  Safety  Institute,  1999).  Human  factors 
refers  to  how  man  and  machine  interact  and  is  taken  into  consideration  when  designing 
products.  A  related  term  is  human  error.  The  Army  defines  human  error  as  “human 
performance  that  deviated  from  that  required  by  the  operational  standards  or  situation 
(Department  of  the  Army,  1999a).  The  Air  Force  uses  various  terms  to  describe  people- 
caused  mishaps:  accepted  risk,  anthropometry,  background,  complacency,  discipline, 
drugs-medicine,  judgment,  pathological,  perceptions,  physiological,  preparation, 
proficiency  and  psychological  (Department  of  the  Air  Force,  1998c:226).  Mishaps 
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attributed  to  human  factors  are  often  deemed  to  be  caused  by  human  error.  Since  human 
error  is  at  least  a  contributing  factor  in  the  majority  of  aviation  mishaps,  it  follows  that 
much  literature  exists  describing  why  human  errors  occur.  Specifically,  research  has 
shown  that  pilot  inexperience,  spatial  disorientation,  and  poor  judgement  are  major 
contributors  to  human  error  accidents. 

Pilot  inexperience  as  a  mishap  cause  factor  has  been  the  subject  of  some  study. 
The  Air  Force’s  Tactical  Air  Command  Class  A  mishap  data  from  1  January  1979  to  31 
December  1983  revealed  that  50  percent  of  operator  caused  mishaps  involved  a  pilot  with 
fewer  than  18  months  and  300  hours  in  the  mishap  aircraft  (McGraw,  1987:5,6). 
Subsequent  changes  to  the  flying  syllabus  focused  on  preventing  inexperienced  pilot 
mishaps.  A  review  of  the  same  data  from  1  January  1984  to  31  December  1986  showed  a 
marked  decrease  in  the  percentage  of  mishaps  involving  inexperienced  pilots.  Similarly, 
Borowsky  showed  that  more  experienced  Naval  pilots  had  a  lower  mishap  rate  than  those 
pilots  with  less  experience.  Experience  was  measured  in  terms  of  total  flying  hours  and 
flying  hours  by  type  of  aircraft  (Borowsky,  1986:ii). 

Spatial  disorientation  is  also  a  common  cause  of  flight  mishaps  and  significant 
research  has  been  done  in  that  area  over  the  years.  Also  called  pilot  vertigo,  spatial 
disorientation  occurs  when  the  pilot  has  an  orientational  illusion  in  flight.  That  is,  he 
believes  he  is  in  a  particular  “position,  attitude,  or  motion  relative  to  the  plane  of  the 
earth’s  surface”  when  in  fact  he  is  not  (Gillingham,  1986:81).  Studies  of  military 
aviation  show  that  the  percent  of  mishaps  in  the  Army  (7.1 1  percent)  and  Navy  (6.75 
percent)  where  spatial  disorientation  was  either  the  cause  or  a  factor  are  consistent  with 
that  of  the  Air  Force.  Studies  during  the  periods  1954-1956,  1964-1967, 1958-1968, 
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1968-1972  and  in  1979  revealed  that  the  percentage  of  Air  Force  mishaps  due  to  spatial 
disorientation  ranged  from  4  to  9.6.  The  data  from  1976  to  1997  reveal  that  average 
spatial  disorientation  Class  A  mishap  rate  for  fighter  aircraft  per  100,000  flying  hours 
was  .39  (or  5.3  percent)  of  all  fighter  Class  A  mishaps  during  that  time  frame  (Air  Force 
Safety  Center,  1999).  The  general  aviation  sector  has  also  suffered  from  spatial 
disorientation  problems  but  not  as  much  as  the  military.  As  of  1979,  only  2.4  percent  of 
general  aviation  mishaps  were  due  to  spatial  disorientation  (1986:83,84). 

The  FAA  has  long  recognized  the  impact  human  error  contributes  toward  causing 
mishaps,  particularly  those  causes  resulting  from  poor  judgement  due  to  a  faulty 
decision-making  process.  As  a  result,  in  1987,  the  FAA  published  a  series  of  aircrew 
training  manuals  entitled  Aeronautical  Decision  Making  (ADM).  In  Aeronautical 
Decision  Making  for  Commercial  Pilots.  Jensen  and  Adrion  cite  four  building  blocks  that 
a  pilot  must  have  for  aviation  safety  and  effectiveness:  knowledge,  skill,  experience,  and 
judgement.  Basic  knowledge  and  skills  are  acquired  through  training  courses,  whereas 
judgement  is  based  on  both  experience  and  training  (Jensen  and  Adrion,  1988:1).  The 
Bell  Jet  Ranger  helicopter,  used  extensively  in  commercial  aviation,  saw  a  significant 
reduction  in  human  error-related  accidents  as  a  result  of  its  crewmembers  using  the  ADM 
training  materials  (Adams,  1992:3).  The  Air  Force  and  Navy  also  saw  significant 
reductions  in  human  error  related  accidents  (Adams,  1992:4)  through  use  of  the  materials. 
It  can  therefore  be  said  that  increased  training,  particularly  in  the  decision  making 
process,  can  have  a  positive  effect  on  human  error  related  mishaps. 

Material  Failure.  The  second  leading  cause  of  aviation  mishaps  is  material 
failure.  Since  humans  make  aircraft  materials,  the  materials  will  eventually  fail. 
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Ultimately  then,  material  failure  can  be  traced  to  a  human  cause.  The  point  of  discussing 
material  failure,  however,  is  to  describe  immediate  reasons  for  the  failure.  The  Army 
defines  materiel  factors  as  “when  materiel  elements  become  inadequate  or  counter¬ 
productive  to  the  operation  of  the  vehicle/equipment/system”  (Department  of  the  Army, 
1999a).  The  Air  Force  uses  various  terms  to  describe  mishaps  due  to  parts:  acquisition, 
attrition,  design,  faulty-part,  modification,  unauthorized  modification,  and  other 
(Department  of  the  Air  Force,  1998c:226-227).  General  causes  of  material  failure 
include  design  not  adequate  for  the  load,  poor  manufacturing  process,  and  material  wear 
out.  Material  wear  out  is  often  due  to  corrosion  or,  in  the  case  of  aircraft  engines,  thermal 
stress. 

Environmental.  Although  not  as  common  a  mishap  cause  as  human  error 
or  material  failure,  the  environment  can  have  a  significant  effect  on  aviation  safety.  The 
Army  identifies  environmental  factors  as  those  conditions  that  affect  human  or  material 
performance.  These  conditions  may  include  weather,  animals,  and  electromagnetic 
environmental  effects  (E3).  E3  result  from  high  intensity  radio  transmissions  and  can 
cause  aircraft  instruments  to  malfunction  (Department  of  the  Army,  1999a). 

Other.  As  stated  earlier,  the  Army  does  not  have  an  “other”  category  but 
the  Air  Force  does.  The  Air  Force  includes  animals,  manning,  other  (new  reason), 
unknown,  and  weather  in  this  category  (Department  of  the  Air  Force,  1998c:227).  The 
Air  Force  does  not  have  on  “environmental”  category  but  animals  and  weather  would 
correspond  to  the  Army's  environmental  category. 

Operations  Tempo.  Operations  tempo  has  been  suggested  as  having  an 
adverse  affect  on  the  mishap  rate.  A  study  performed  by  the  Air  Force  in  1994  found  no 
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direct  correlation  between  the  two  (General  Accounting  Office,  1996:2).  A  year  later, 
however,  the  Air  Force  Chief  of  Staff  commissioned  four  retired  flag  officers,  a  Blue 
Ribbon  Panel,  to  examine  the  Air  Force  safety  program.  The  Panel  did  not  define  the 
term  “operational  tempo”  or  express  measures  of  how  it  is  determined.  Nevertheless,  it 
determined  that  at  least  six  factors  within  the  control  of  the  Air  Force  contributed  to 
increased  operations  tempo  and  presented  an  aviation  safety  risk:  organizational 
changes;  an  unwritten,  implicit  masters  degree  requirement;  personnel  policies;  stress  at 
the  operational  level  due  to  frequent  deployments;  two  level  maintenance 
implementation;  support  equipment  shortages  (Department  of  the  Air  Force,  1995:19,20). 

Mishap  Prevention  Factors.  For  every  mishap,  there  is  a  possible  way  that  it 
could  have  been  prevented.  This  section  considers  four  major  areas  of  mishap  prevention 
as  described  in  the  literature:  leadership,  mishap  investigation,  advancement  in 
technology,  and  human  factor  programs.  Figure  2  depicts  these  factors. 

Leadership 

41 


Human  Factor  Programs 
Figure  2.  Aviation  Mishap  Prevention  Factors 
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Leadership.  According  to  the  Defense  Science  Board,  “leadership  is  the 
single  most  important  factor  affecting  aviation  safety;  Commanders  at  every  level  must 
be  personally  involved”  (Department  of  Defense,  1997:9).  It  would  be  beyond  the  scope 
of  this  literature  review  to  pursue  a  discourse  on  the  Army  and  Air  Force  philosophies  on 
leadership.  One  can  assume  that  there  is  no  difference  between  what  the  Army  and  the 
Air  Force  think  makes  a  good  military  leader.  The  more  appropriate  discussion  is  the 
emphasis  the  two  services  place  on  aviation  safety  based  on  where  that  responsibility 
rests  in  the  chain  of  command. 

Army.  All  aviation  personnel  have  responsibility  for  mishap 
prevention  but  the  Army  identifies  specific  positions  whose  occupants  have  explicit 
responsibility  for  aviation  safety.  The  Secretary  of  the  Army  and  his  advisor,  the 
Assistant  Secretary  of  the  Army  for  Installations,  Logistics,  and  Environment,  have 
responsibility  at  the  Headquarters  level.  The  commander  of  the  Army  Safety  Center 
oversees  the  Army  Safety  Program  and  the  Director  of  Army  Safety  is  in  charge  of  the 
Aviation  Accident  Prevention  Program.  Commanders  at  all  levels  are  responsible  for 
aviation  safety  under  their  command.  Advising  the  commander  on  safety  issues  is  the 
unit  aviation  safety  officer,  the  flight  surgeon,  and  the  aviation  maintenance  officer  who 
is  responsible  for  an  aircraft  preventive  maintenance  program.  The  aviator  is  at  the  core 
of  aviation  accident  prevention.  At  the  installation  and  major  command  (MACOM) 
levels,  the  command  safety  director  is  responsible  for  the  management  of  the  command 
safety  program  and  advises  the  MACOM  commander  on  aviation  safety  issues 
(Department  of  the  Army,  1999b). 
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Air  Force.  Like  the  Army,  the  Air  Force  has  designated  positions 
and  organizations  responsible  for  safety  as  described  in  Air  Force  Policy  Directive  91-2. 
The  Air  Force  Chief  of  Safety  sets  safety  policy  and  guidance.  The  Assistant  Secretary 
of  the  Air  Force  for  Manpower,  Reserve  Affairs,  Installations,  and  Environment  oversees 
occupational  safety  and  health  policy  issues.  Commanders  must  provide  a  safe  working 
environment  by  ensuring  a  mishap  prevention  program  is  in  place.  The  safety  focus  of 
the  commander  is  the  prevention  of  mishaps  both  on  and  off  duty  and  incorporating 
safety  principles  to  accomplish  the  mission.  All  Air  Force  personnel  are  responsible  for 
safety  by  identifying  and  reporting  hazards  up  the  chain  of  command.  Safety  staffs  at 
various  levels  manage  the  commander’s  mishap  prevention  program.  This  includes  the 
safety  office  at  wing  level  and  the  flight  safety  officer  at  the  flying  unit  level.  The  Air 
Force  Safety  Center  has  a  wide  range  of  responsibilities  including  mishap  investigation, 
data  storage,  tracking  corrective  actions,  and  evaluating  unit  safety  programs 
(Department  of  the  Air  Force  1993:2,3).  Table  1  summarizes  the  Army-Air  Force 
leadership  comparison. 


Table  1 .  Army  and  Air  Force  Safety  Responsibilities 


Level 

Army 

Air  Force 

Headquarters 

Secretary  of  the  Anny 

Assistant  Secretary  of  the 

Army  for  Installations, 

Logistics  and  Environment 

Assistant  Secretary  of  the  Air 

Force  for  Manpower,  Reserve 
Affairs,  Installations,  and 
Environment 

Safety  Center 

Director  of  Army  Safety 

Air  Force  Chief  of  Safety 

Major  Command 

Safety  Director 

Safety  Staffs 

Unit 

Commanders 

Aviation  Safety  Officers 
Individuals 

Commanders 

Flight  Safety  Officers 

Individuals 
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The  Air  Force  has  long  realized  the  importance  of  the  commander’s  role  in 
aviation  safety.  In  1987,  the  Tactical  Air  Command  (TAC)  published  the  Commander’s 
Guide  for  the  Prevention  of  TAC  Fighter  Mishaps.  A  review  of  mishaps  within  the 
command  revealed  that  supervisory  error  was  causal  or  contributed  to  35%  of  the 
mishaps  between  January  1980  and  December  1985  (McGraw,  1987:1).  The  guide 
addresses  flying  safety  issues  new  squadron  and  wing  commanders  will  face  when  they 
begin  their  duties.  It  specifically  addresses  the  fact  that  a  burden  of  responsibility  for  the 
safety  and  welfare  of  those  under  his  command  is  placed  upon  hini.  As  then  TAC 
commander  General  Russ  stated,  the  commander  is  not  “one  of  the  boys”  anymore,  he  is 
“they”  (1987:3).  He  is  the  leader  and  must  lead  by  example  in  all  aspects  of  life.  The 
guide  emphasizes  that  new  commanders  study  and  learn  from  the  mistakes  of  previous 
commanders.  It  also  makes  clear  that  the  principles  and  issues  addressed  are  not 
exclusive  to  the  fighter  community  but  have  application  in  other  major  commands  and 
the  nonfighter  aircraft  community. 

Mishan  Investigation.  Aircraft  mishap  investigation  has  been  performed 
ever  since  the  Wright  brothers  suffered  the  crash  of  their  aircraft  at  Fort  Meyer,  VA  in 
1 909.  It  has  long  been  noted  within  the  aviation  community  that  accurate  and  timely 
mishap  investigations  can  help  prevent  similar  future  mishaps.  Over  the  years,  the  most 
effective  way  to  prevent  mishaps  has  been  through  the  mishap  investigation  and  reporting 
process.  Both  in  civil  and  military  aviation,  a  safety  board  convenes  to  find  out  what 
caused  a  mishap  and  why  and  disseminates  that  information  to  help  prevent  similar 
mishaps.  The  information  goes  to  aircraft  design  and  acquisition  personnel  who  can  then 
incorporate  modifications  and  safety  features  into  the  aircraft.  The  information  also  goes 
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out  to  current  users  of  the  aircraft  so  that  they  can  make  any  necessary  changes  to  their 
procedures. 

In  civilian  aviation,  the  Federal  Aviation  Administration  (FAA)  oversees  the 
commercial  and  general  aviation  activities.  When  mishaps  occur,  however,  the  National 
Transportation  Safety  Board,  an  agency  independent  from  the  FAA,  investigates  to 
determine  the  cause  and  provides  recommendations  to  the  FAA  for  corrective  action. 
Since  “public  safety  is  the  first  priority  in  civil  aviation,”  this  independence  has  worked 
well  in  terms  of  mishap  rate  reduction  (Department  of  the  Air  Force,  1995:6). 

In  the  military,  each  service  has  its  own  process  for  mishap  investigation.  Studies 
have  been  done  to  determine  whether  an  independent  investigative  board  should  be 
established  within  DoD.  One  of  the  congressional  taskings  for  the  Defense  Science 
Board  (DSB)  was  to  “determine  the  need/value  of  a  joint  program  to  require  a 
standardized  process  for  reporting  and  assessing  the  causes  of  accidents”  (Department  of 
Defense,  1997:4).  The  DSB  found  that  each  service  had  its  own  appropriate  process  for 
investigating,  reporting,  and  assessing  mishap  causes.  Although  each  service  organized 
and  administered  investigations  differently,  each  service  incorporated  basic  mishap 
investigation  principles  adequately  and  therefore  there  would  be  no  added  value  or  need 
for  a  joint  program  (1997:16,17).  In  its  study  of  the  same  issue,  the  Blue  Ribbon  Panel 
concluded  that  since  the  first  priority  for  the  military  is  combat  efficiency,  it  would  not  be 
in  the  military’s  best  interest  to  have  a  separate  investigative  arm  (Department  of  the  Air 
Force,  1995:6,7). 

Advancement  in  Technology.  In  the  early  days  of  aviation,  advancement 
in  technology  made  major  contributions  to  reducing  the  mishap  rate.  As  aircraft  and  their 
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components  have  become  more  reliable,  the  proportion  of  mishaps  due  to  poor  design  or 
manufacture  has  decreased  while  the  human  error  proportion  has  increased  (Driskell  and 
Adams,  1992:3).  Nevertheless,  technology  continues  to  advance,  opening  new 
opportunities  to  help  further  reduce  the  proportion  of  mishaps  due  to  aircraft  failure.  In 
its  report,  the  DSB  recommended  that  the  DoD  provide  funding  for  high  priority  flight 
safety  equipment  as  well  as  for  research,  design,  development,  test,  and  deployment  of 
new  equipment.  Furthermore,  the  DoD  should  use  information  technology  to  develop  a 
joint  database  of  flight  data  recorder  performance  and  maintenance  information.  This 
shared  resource  could  help  prevent  accidents  through  lessons  learned  from  each  of  the 
services  (Department  of  Defense,  1997:ES-4,5). 

Human  Factors  Programs.  Since  human  error  is  the  leading  contributing 
factor  to  the  majority  of  DoD  aviation  mishaps,  the  service  leaders  have  increased 
emphasis  on  programs  that  aim  to  minimize  human  error.  Two  such  programs  are  crew 
resource  management  (CRM)  and  risk  management  (RM). 

Crew  Resource  Management.  The  definition  of  CRM  is  “the 
effective  utilization  of  all  available  resources  -  equipment  and  people  -  to  achieve  safe, 
efficient  flight  operations”  (Driskell,  1992:8).  Although  military  and  commercial  efforts 
to  heighten  safety  emphasis  through  CRM  have  been  ongoing  since  the  1970s,  a  CRM 
training  program  was  not  established  in  the  Air  Force  until  1994.  In  its  investigation,  the 
Blue  Ribbon  Panel  found  that  CRM  has  been  beneficial  to  the  multi-member  cockpits, 
but  its  impact  in  the  single  seat  fighter  environment,  where  most  of  the  Class  A  mishaps 
occur,  is  unclear  (Department  of  the  Air  Force,  1995:14). 
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Risk  Management.  “Risk  management  is  the  process  of 
identifying,  assessing,  and  controlling  risks  arising  from  operational  factors  and  making 
decisions  that  balance  risk  costs  with  mission  benefits”  (Department  of  the  Army, 

1998a:  1-1).  It  is  a  tool  for  the  military  leader  to  use  to  minimize  risks  and  mavimizft 
mission  accomplishment.  Although  the  military  has  practiced  managing  risks  for  years,  it 
has  done  so  based  on  intuition  and  experience  (Department  of  the  Air  Force,  1998:4). 
While  the  concept  of  managing  risks  is  not  new  to  the  military,  RM  formalizes  that 
process.  The  DSB  recommended  that  the  services  make  full  use  of  risk  management 
practices  as  they  provide  the  least  costly  means  of  reducing  the  mishap  rates  (Department 
of  Defense,  1997:31,32). 

Summary.  This  section  described  the  aviation  safety  factors  pertinent  to  the 
research.  It  presented  both  mishap  cause  and  mishap  prevention  factors  as  found  in  a 
review  of  the  literature  and  described  how  those  factors  related  to  Army  and  Air  Force 
aviation  safety.  The  mishap  cause  factors  were:  human  factors,  material  failure, 
environmental,  and  other.  The  mishap  prevention  factors  were:  leadership,  mishap 
investigation,  technological  advances,  and  human  factor  programs.  Two  human  factor 
programs,  CRM  and  RM,  were  discussed  but  remainder  of  this  literature  review  focuses 
on  risk  management  history,  defines  terms,  and  compares  the  Army  and  Air  Force 
implementation  of  risk  management. 

Risk  Management  History 

The  roots  of  risk  management  are  embedded  in  the  science  of  uncertainty: 
probability.  The  17th  century  French  mathematicians,  Blaise  Pascal  and  Pierre  De 
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Fermat,  are  considered  the  founders  of  the  study  of  probability  (Barnett,  1996:363). 

Since  risk  is  the  probability  of  a  loss,  risk  management  has  been  practiced  in  several 
industries  that  involve  high  risk. 

Insurance.  “The  risk  at  any  given  time  equals  the  difference  between  the  reserve 
and  the  face  of  the  policy”  (Neilson,  1958).  The  insurance  industry  uses  risk 
management  concepts  to  evaluate  whether  to  offer  insurance  for  someone,  and  if  so,  what 
it  will  cost  the  insured.  The  higher  the  risk  to  the  company,  the  higher  the  cost  will  be  for 
the  insured. 

Transportation.  The  transportation  industry  uses  risk  management  principles  to 
determine  routes  when  hauling  hazardous  materials.  The  goal  is  to  transport  hazardous 
materials  as  quickly  and  as  safely  as  possible  in  order  to  minimize  the  risk  of  exposure  to 
the  environment.  Factors  considered  include  type  and  quantity  of  material,  mode  of 
transport,  starting  point,  destination,  and  population  density.  Models  have  been 
developed  that  assign  weight  to  the  various  factors  to  help  decide  the  route  (Helander, 
1997:216-226). 

Government.  The  National  Aeronautics  and  Space  Administration  (NASA)  has 

integrated  risk  management  into  their  strategic  planning.  Historically,  NASA  has  used 

reliability  engineering  principles  to  curb  risks.  Ebling  defines  reliability  engineering. 

The  overall  objective  of  reliability  engineering  is  to  ensure  that  the  final 
product  will  be  both  economically  reliable  and  reliably  safe.... Reliably 
safe  requires  designing  sufficient  reliability  into  the  product  to  ensure  that 
the  probability  of  accidents,  injury,  or  death  resulting  from  a  product 
failure  is  within  an  acceptable  limit.  (Ebling,  1997:429-430) 

RM  is  a  decision-making  process  based  on  a  cost-benefit  analysis  of  known  hazards  and 

desired  objectives.  Reliability  engineering,  however,  is  an  engineering  process  based  on 
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the  probability  of  design  failure.  An  RM  decision-maker  should  take  into  account  any 
reliability  information  available  to  make  the  best  decision.  While  not  minimi?ing 
reliability  engineering  techniques,  NASA  is  now  focusing  efforts  on  managing  the  risks 
of  systems  interaction.  NASA’s  motivation  for  mission  success  stems  from  the  fact  that, 
like  the  military,  its  budget  is  decreasing  yet  more  missions  are  demanded  (Lalli, 
1996:355).  NASA  must  determine  ways  of  ensuring  reliable  missions.  The  answer  is 
through  identifying  the  hazards  and  mitigating  the  risks  associated  with  those  hazards, 
risk  management. 

The  background  of  risk  management  has  been  presented,  but  before  discussing  the 
specifics  of  RM  in  the  Army  and  Air  Force,  some  terms  need  to  be  defined. 

Definitions 

Risk.  Since  this  discussion  regards  both  the  Army  and  Air  Force,  it  is  important 
to  define  the  terminology  that  will  be  used  throughout.  At  the  heart  of  military  aviation 
mishap  reduction  and  risk  management  is  the  term  risk.  Webster’s  dictionary  defines  risk 
as  a  “hazard;  danger;  peril;  exposure  to  loss,  injury,  disadvantage,  or  destruction” 
(Neilson,  1958).  The  Army  defines  risk  as  the  “chance  of  hazard  or  bad  consequences; 
the  probability  of  exposure  to  chance  of  injury  or  loss  from  a  hazard;  risk  level  is 
expressed  in  terms  of  hazard  probability  and  severity”  (Department  of  the  Army,  1998a). 
The  Air  Force  defines  risk  as  “an  expression  of  consequences  in  terms  of  the  probability 
of  an  event  occurring,  the  severity  of  the  event  and  the  exposure  of  personnel  or  resources 
to  potential  loss  or  harm”  (Department  of  the  Air  Force,  1998a:37).  Both  the  Army  and 
Air  Force  emphasize  the  risk  elements  of  probability,  severity,  and  exposure.  Despite 
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minor  wording  differences,  there  is  no  significant  difference  in  definitions  of  risk 
between  the  services.  All  three  definitions  are  in  general  agreement. 

Risk  Management.  Although  there  is  minimal  DoD-level  risk  management 
guidance  for  the  services.  Joint  Pub  3-0,  Doctrine  for  Joint  Operations  and  Joint  Pub  5-0, 
Doctrine  for  Planning  Joint  Operations,  do  allude  to  the  practice  of  risk  management 
principles.  Since  each  service  is  responsible  for  its  own  safety  program  and  safety  and 
risk  are  closely  related,  each  service  has  developed  its  own  guidelines  and  instructions  for 
governing  its  risk  management  programs.  The  Army  uses  the  term  risk  management 
(RM)  and  defines  it  as  “the  process  of  identifying,  assessing,  and  controlling  risks  arising 
from  operational  factors  and  making  decisions  that  balance  risk  cost  with  mission 
benefits”  (Department  of  the  Army,  1998a:G-3).  The  Air  Force  uses  the  term  operational 
risk  management  (ORM)  and  defines  it  as  “a  logic-based,  common  sense  approach  to 
making  calculated  decisions  on  human,  materiel,  and  environmental  factors  before, 
during  and  after  Air  Force  mission  activities  and  operations”  (Department  of  the  Air 
Force,  1997:1).  The  term  used  in  this  paper  (RM  vs.  ORM)  depends  on  which  service  is 
under  discussion.  There  is  essentially  no  difference  between  these  definitions  in  that  they 
both  state  that  risk  management  is  a  decision  making  process. 

RM  Principles.  The  Army's  RM  principles  are  1)  accept  no  unnecessary  risks,  2) 
make  risk  decisions  at  the  appropriate  level  to  establish  clear  accountability,  and  3) 
accept  risk  when  benefits  outweigh  the  costs  (Department  of  the  Army,  1998:1-3).  The 
Air  Force  adds  a  fourth  principle:  integrate  ORM  into  Air  Force  doctrine  and  planning  at 
all  levels  (Department  of  the  Air  Force,  1998a:6). 
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RM  Process.  The  Army's  RM  process  consists  of  five  steps:  1)  identify  hazards, 
2)  assess  hazards  to  determine  risks,  3)  develop  controls  and  make  risk  decision,  4) 
implement  controls,  5)  supervise  and  evaluate  (Department  of  the  Army,  1998a:2-0). 

The  only  difference  between  the  Army  and  Air  Force  risk  management  process  is  that  the 
Air  Force  splits  the  Army’s  third  step  into  two  steps.  Consequently,  the  Army  uses  a 
five-step  process  and  the  Air  Force  uses  a  six-step  process. 

DODI  6055.7  defines  the  following  flight  mishap  terms: 

Flight  and  Flight-Related  Mishap.  An  aircraft  flight  or  flight-related  mishap 
occurs  when  intent  to  fly  exists.  Specifically,  a  flight  mishap  occurs  when  “there  is 
reportable  damage  to  the  aircraft  itself.”  A  flight-related  mishap  occurs  when  there  is  “no 
reportable  damage  to  the  aircraft  itself,  but  the  mishap  involves  fatality,  injury... or  other 
property  damage.”  Until  1983,  the  Army  flight  mishap  rate  was  calculated  based  on  both 
flight  and  flight-related  mishaps.  Beginning  in  1984,  the  Army  flight  mishap  rate  was 
calculated  based  on  flight  mishaps  only. 

Intent  for  Flight.  Intent  for  flight  is  assumed  during  the  time  from  takeoff  brake 
release  or  power  application  until  landing  is  completed. 

Mishap  Categories.  Mishaps  are  categorized  according  to  their  severity  in  terms 
of  dollar  value  of  damage  and  personal  injury.  A  Class  A  mishap  involves  over  $1 
million  in  property  damage,  a  destroyed  aircraft,  or  loss  of  life  or  permanent  total 
disability.  A  Class  B  mishap  involves  over  $200,000  but  less  than  $1,000,000  in  property 
damage  or  permanent  partial  disability  or  when  five  or  more  personnel  are  inpatient 
hospitalized.  A  Class  C  mishap  involves  over  $10,000  but  less  than  $200,000  in  property 


20 


damage  or  injury  causing  loss  of  time  from  work  beyond  the  day  or  shift  upon  which  it 
occurred. 

Mishap  Rate.  The  mishap  rate  is  the  number  of  mishaps  per  100,000  flight  hours. 

Risk  Management  Implementation 

A  hypothesis  of  this  research  is  that  the  Air  Force  implementation  of  ORM  is 
significantly  different  from  the  Army's  implementation  of  RM.  If  this  is  not  true,  and  the 
Army  aviation  mishap  rate  saw  a  significant  reduction  due  to  RM  application,  then  the 
Air  Force  may  also  see  a  similar  reduction  in  its  mishap  rate.  Implementation  of  RM  into 
the  Army  and  of  ORM  into  the  Air  Force  was  evaluated  using  three  criteria:  published 
directives,  responsibility,  and  training. 

Army.  The  Army  officially  began  the  implementation  of  RM  in  1987.  AR  385- 
10,  The  Army  Safety  Program,  addressed  the  integration  of  risk  management  throughout 
the  Army:  “Decision-makers  at  every  level  will  employ  risk  management  approaches  to 
effectively  preclude  unacceptable  risk  to  the  safety  of  personnel  and  property” 
(Department  of  the  Army,  1999c).  According  to  Steams,  RM  was  “successfully 
integrated  into  the  Army’s  training  and  operational  process”  (Steams,  1990:32). 

In  1997,  the  Secretary  of  the  Army  formally  established  responsibilities  for  the 
full  integration  of  RM  (Department  of  the  Army,  1998b).  At  first  RM  was  a  safety 
officer  function  applicable  to  the  training  and  operational  areas,  particularly  the  aviation 
community  (Department  of  the  Army,  1998a:iii).  However,  FM  100-14,  Risk 
Management  the  Army  manual  for  the  application  of  RM,  clearly  places  the 
responsibility  of  RM  integration  on  commanders,  leaders,  staffs,  and  individuals.  Further 
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RM  integration  guidance  stipulates  that  “HQDA  (Headquarters,  Department  of  the  Army) 
Principal  Officials  and  MACOM  (major  command)  commanders  are  designated 
integrating  agents”  (Army  Safety  Center,  1999a). 

Risk  management  was  incorporated  into  the  Army's  material  acquisition  process 
in  the  late  1980's  and  into  its  doctrine,  training,  and  professional  military  education  in  the 
early  1990's  (V an  Aalten,  1999).  The  Army  attributes,  in  part,  its  recent  aviation  mishap 
reduction  success  to  the  aviation  safety  officers  who  applied  RM  techniques  (Army 
Safety  Center,  1999b).  However,  in  the  Army's  recent  integration  effort,  it  has 
recognized  the  need  for  a  cadre  of  trained  safety  personnel  skilled  in  being  able  to 
broadly  apply  RM  techniques  (Army  Safety  Center,  1999b).  The  goal  is  to  integrate  RM 
practice  and  training  for  all  individuals  both  on-  and  off-  duty  in  order  that  it  become 
second  nature  and  embedded  in  the  Army  culture  (Department  of  the  Army,  1998a:iii). 

Air  Force.  On  2  Sep  96,  the  Air  Force  Chief  of  Staff  ordered  the  implementation 
of  ORM  to  begin.  Full  implementation  through  computer-based  awareness  training  on  an 
individual  level  was  completed  1  Oct  98.  Thus,  Air  Force  ORM  is  in  its  infancy.  AFI 
91-213,  “Operational  Risk  Management  (ORM)  Program,”  established  the  requirement  to 
incorporate  ORM  programs  throughout  the  Air  Force  and  outlines  their  description, 
management,  and  development.  Each  major  command  is  responsible  for  developing  and 
implementing  its  own  ORM  programs.  AFPAM  91-215,  “Operational  Risk  Management 
(ORM)  Guidelines  and  Tools,”  describes  how  to  integrate  and  execute  the  ORM  process. 

While  it  is  the  responsibility  of  the  major  commands  to  develop  and  implement  its 
own  programs,  it  is  the  responsibility  of  commanders  at  all  levels,  supervisors,  and 
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individuals  to  execute  the  practice  of  ORM  both  on-  and  off-  duty  (Department  of  the  Air 
Force,  1998a:4). 

While  the  Army’s  initial  emphasis  was  to  apply  RM  through  the  commanders  in 
the  operational  and  training  areas,  from  the  start  the  Air  Force  took  a  more  holistic 
approach  with  application  in  all  areas.  The  emphasis  in  the  Air  Force  has  been  the 
implementation  of  ORM  through  training  and  education  from  the  top  down.  There  are 
four  levels  of  training:  awareness,  mission  and  workplace  specific,  supervisor,  and  ORM 
advisor.  The  Air  Force  provides  training  to  a  cadre  of  major  command  safety 
professionals  who,  in  turn,  provide  training  and  guidance  at  the  wing  levels.  ORM  has 
also  begun  to  be  taught  in  the  initial  stages  of  an  individual’s  career  beginning  with  basic 
military  training  and  officer  commissioning  sources.  It  is  also  taught  at  the  various 
technical  schools  and  professional  military  education  courses  throughout  one’s  career 
(Department  of  the  Air  Force,  1997:4-5). 

While  the  Army  and  Air  Force  have  taken  different  tracks  to  the  same  goal,  it 
does  not  appear  that  their  risk  management  implementations  have  been  significantly 
different.  Both  services  have  established  a  trained  cadre  of  risk  management  personnel 
and  are  integrating  the  philosophy  at  all  levels  for  all  individuals  to  apply  both  on-  and 
off-  duty.  It  is  likely  that  the  Air  Force  learned  much  from  the  Army  before  launching  its 
ORM  program. 

Summary.  This  section  examined  the  implementation  of  RM  in  the  Army  and 
ORM  in  the  Air  Force  using  three  criteria:  published  directives,  responsibility,  and 
training.  The  Army  began  implementation  in  1987  and  focused  its  implementation 
efforts  in  the  acquisition,  operational,  and  training  areas.  It  later  expanded  RM 
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implementation  into  its  doctrine  and  educational  process.  The  Air  Force  began 
implementation  in  1996  and  completed  awareness  training  in  1998.  From  its  inception, 
ORM  was  implemented  into  all  areas.  While  RM  implementation  has  progressed  at 
different  rates  for  each  service,  both  the  Army  and  Air  Force  are  taking  similar  steps  to 
arrive  at  the  same  goal:  mishap  prevention,  both  on-  and  off-  duty.  Thus,  the 
implementation  procedures  do  not  appear  to  be  significantly  different. 

Summary  and  Conclusions 

This  chapter  provided  a  review  of  the  aviation  safety  factors  as  revealed  in  the 
literature,  including  those  related  to  mishap  cause  and  prevention.  A  history  of  risk 
management  was  discussed  as  well  as  the  definitions  of  risk  management  terms.  How  the 
Army  and  Air  Force  have  implemented  risk  management  concepts  into  their  operations 
was  examined.  Based  on  this  review,  it  appears  that  the  focus  of  aviation  mishap 
reduction  is  centered  on  the  application  of  risk  management.  The  Army  has  put  much 
effort  into  reducing  its  mishaps  through  RM  techniques  and  its  mishap  rate  has  declined. 
Was  this  decline  due  to  RM  application?  The  next  chapter  will  describe  how  the 
available  data  was  analyzed  and  may  help  answer  this  question. 
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ICE.  Methodology,  Results,  and  Analysis 


Overview 

This  chapter  is  the  heart  of  the  research  project.  It  describes  the  data  collected 
and  reviews  the  possible  sources  of  time  series  invalidity.  The  primary  focus  of  this 
chapter,  however,  is  a  presentation  of  die  methodology,  results,  and  analysis  that  address 
the  investigative  questions  posed  in  Chapter  I. 

Data  Collection  and  Description 

To  conduct  the  analysis,  aviation  safety  data  was  acquired  from  both  the  Army 
and  Air  Force. 

Army.  The  Army  Safety  Center  (ASC)  provided  Class  A  and  Class  A-C  data 
(rates  and  numbers  of  mishaps)  from  1973  to  1998  but  the  individual  Class  B  and  C  data 
were  not  available.  Additionally,  Class  A,  B,  and  C  data  by  aircraft  type  from  1988-1998 
were  provided.  By  extracting  the  Class  A  data  out  of  the  Class  A-C  data,  a  separate 
category  of  Class  B-C  mishaps  was  constructed.  The  ASC  also  provided  flight  mishap 
causal  data  for  fiscal  years  1990  to  1998.  The  data  include  aircraft  type,  class,  and  cause 
category  (human  error,  materiel  failure,  and  environmental)  for  each  year.  Since  Class 
A-C  mishap  rates  include  events  in  the  B-C  category,  only  Class  A  and  Class  B-C 
category  mishap  rates  were  analyzed.  Table  2  summarizes  the  Army  data  collected. 

The  mishap  rates  prior  to  and  including  1983  were  computed  based  on  flight  and  flight- 
related  mishaps.  Since  1984,  however,  rates  have  been  computed  based  on  flight  mishaps 
only.  The  mishap  rate  based  on  flight  and  flight-related  mishaps  will  be  higher  than  if 
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Table  2.  Army  Data  Summary 


FY 

Class  A  Rate 

Class  A-C  Rate 

ByMTDS 

#  of  Mishaps 

Causes 

73-87 

YES 

YES 

NO 

YES 

(Class  A  only) 

NO 

88-98 

YES 

YES 

YES 

YES 

NO 

90-98 

YES 

YES 

YES 

YES 

YES 

flight  mishaps  only  are  considered  in  the  calculation.  This  confound  is  addressed  later  in 
this  chapter. 

Since  the  Army  implemented  RM  in  1987,  the  data  cover  a  period  before  and 
after  implementation  although  causal  information  dates  back  only  to  1990.  The  Army 
data  is  presented  in  Appendix  A. 

Air  Force.  The  Air  Force  Safety  Center  provided  flight  mishap  causal  data  for 
fiscal  years  1993  to  1998.  Since  the  Air  Force  completed  initial  ORM  implementation  in 
1998,  the  data  only  cover  the  period  before  implementation.  The  data  include  aircraft 
type,  accountable  category,  responsible  agent,  and  reason  for  each  year.  As  described  in 
Chapter  II,  the  reasons  are  further  broken  down  into  people,  parts,  paper,  and  other 
categories.  The  people  category  is  of  interest  to  this  research  and  corresponds  to  the 
Army’s  human  error  category.  Class  A  data  from  1947-1997  and  Class  A  and  B  data  by 
aircraft  type  from  1972-1998  were  obtained  from  the  Air  Force  Safety  Center  World 
Wide  Web  site.  The  Air  Force  data  and  related  information  are  presented  in  Appendix  B. 

Sources  of  Time  Series  Invalidity 

This  research  treats  the  mishap  rate  as  the  response  variable  and  risk  management 
as  the  treatment.  Given  that  time  series  data  is  evaluated  in  this  research,  care  must  be 
taken  to  assess  the  effect  of  threats  to  internal  and  external  validity.  Campbell  and 
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Stanley  describe  various  types  of  experimental  designs  and  identify  the  associated 
sources  of  threat  to  their  internal  and  external  validity  (Campbell,  1963).  A  more 
rigorous  discussion  of  the  threats  is  found  in  Cooper  and  Emory  (1995:357-361).  This 
section  specifically  identifies  the  threats  to  the  evaluation  of  aviation  mishap  rate  data  as 
it  relates  to  the  implementation  of  risk  management  in  the  Army  and  Air  Force. 

Internal  Validity.  Campbell  and  Stanley  identify  eight  potential  threats  to  internal 
validity  for  a  time  series  design  study  (Campbell,  1963:40).  Only  the  failure  to  control 
history,  however,  is  considered  a  strong  threat  relative  to  the  other  seven  sources. 

History.  For  a  time  series  study,  history  is  the  single  largest  threat  to 
internal  validity.  It  is  possible  that  factors  other  than  that  being  tested  caused  a  change  to 
occur.  As  stated  in  Chapter  II,  there  are  numerous  reasons  for  an  aviation  mishap  and  at 
least  as  many  ways  of  prevention.  Four  major  aviation  mishap  prevention  factors  were 
identified  in  Chapter  II:  leadership,  mishap  investigation,  advancement  in  technology, 
and  crew  resource  management,  a  human  factors  program.  Risk  management,  also  a 
human  factors  program,  is  only  one  way  of  preventing  mishaps  and  it  may  or  may  not 
have  been  a  factor  in  any  significant  change  in  the  Army's  aviation  mishap  rate  in  recent 
years.  These  mishap  prevention  factors  are  together  considered  responsible  for  any  trend 
in  the  mishap  rate  before  1987.  If  there  were  any  changes  in  the  trend  after  1987,  it 
would  first  have  to  be  attributed  to  any  identifiable  changes  in  those  factors  if  they  were 
contemporaneous  with  RM  (1986-1988).  After  that,  any  change  left  over  can  be 
attributed  to  the  treatment  of  RM.  A  brief  historical  overview  of  each  mishap  prevention 
factor  was  conducted  to  examine  any  confounds  that  may  exist. 
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Leadership  History.  The  Army  recognizes  that  aviation  safety  can 


be  improved  through  proper  leadership.  In  1984  the  Director  of  Army  Safety  became  a 
general  officer  position  to  report  directly  to  the  Army  Chief  of  Staff  (Hicks,  1 999). 
Additionally,  the  Army  has  conducted  much  research  to  continuously  improve  the 
leadership  development  of  its  members.  In  1986,  the  Noncommissioned  Officer 
Professional  Development  Study  was  conducted.  In  1987,  the  Army  Chief  of  Staff 
directed  that  the  Deputy  Commandant,  Command  and  General  Staff  College  (CGSC) 
conduct  a  comprehensive  leader  development  study.  The  Leader  Development  Support 
System,  in  use  today,  was  established  in  1989  as  a  direct  result  of  the  CGSC  study.  In 
1991  the  Leader  Development  Investment  Strategy  study  was  performed  to  address  issues 
facing  the  rapidly  downsizing  Army  (Department  of  the  Army,  1999d).  Currently,  the 
Army  Safety  Center  is  leading  the  Aviation  Safety  Investment  Strategy.  This  effort  is 
examining  mishaps  by  aircraft  type  and  is  developing  strategies  to  reduce  mishaps 
(Hicks,  1999). 

Mishap  Investigation  History.  Mishap  investigation  has  also  been 
considered  to  improve  aviation  safety.  In  its  continuing  effort  to  prevent  aircrew  injuries, 
the  Army's  Aeromedical  Research  Laboratory  (AARL)  established  the  Aviation  Life 
Support  Equipment  Retrieval  Program  (ALSERP)  in  1972.  Members  from  the  AARL 
often  participate  in  mishap  investigations.  By  retrieving  and  evaluating  equipment 
involved  in  mishaps,  the  AARL  provides  critical  design  criteria  for  improved  life  support 
equipment.  Specifically,  three  types  of  helmets  have  been  fielded  since  1972:  the  SPH-4 
Mod  (1978),  the  SPH-4B  (1991),  and  the  HGU-56/P  (1995).  “Each  of  these  helmets 
incorporate  incremental  improvements  designed  or  intended  to  reduce  or  prevent  injury” 
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(Voisine,  1996:3).  Flight  data  recorders  are  useful  in  mishap  investigation  because  their 
data  can  be  downloaded  into  a  computer  and  the  mission  can  be  “flown”  again. 
Investigators  can  then  determine  what  the  aircraft  was  doing  before  and  during  the 
mishap  sequence.  In  1986,  the  UH-60A  Black  Hawk  was  the  first  Army  aircraft  have 
flight  data  recorders  installed  (Army  Weaponry  and  Equipment,  1986:387).  Flight  data 
recorder  installation  is  ongoing  today  (Hicks,  1999).  Changes  in  the  composition  of 
mishap  investigation  teams  have  also  taken  place.  In  the  late  1980’s  the  Army  Safety 
Center  representation  on  mishap  investigations  dropped  from  three  to  two.  The  civilian 
safety  specialist  was  removed  due  to  resource  constraints  (Hicks,  1999).  In  the  mid- 
1990’s,  the  ASC  information  requirement  for  Class  C  mishap  reporting  was  reduced  to 
the  Abbreviated  Aviation  Accident  Report.  This  policy  is  currently  being  re-evaluated  to 
ensure  that  all  necessary  data  is  archived  (Hicks,  1999). 

Technology  Advancement  History.  Technology  advancements 
have  made  Army  aviation  relatively  safer  over  the  years.  Fielding  new  or  modified 
aircraft  with  improved  safety  features  might  contribute  toward  mishap  reduction.  In  1984 
the  first  CH-47D  Chinook  was  delivered.  It  is  an  upgrade  from  the  A,  B,  and  C  models 
(Army  Weaponry  and  Equipment,  1986:392).  The  UH-60A  Black  Hawk  helicopter  was 
fielded  between  1978  and  1989.  The  aircraft  include  crashworthy  armored  crew  seats 
(1986:387).  In  1986  the  first  AH-64A  Apache  attack  helicopter  was  delivered  to  Ft. 
Hood,  TX  (1986:387).  Also  in  1986,  the  first  modernized  AH-1S  Cobra  was  delivered 
through  the  Cobra  Fleet  Life  Extension  Program.  One  if  the  program's  key  features  was  a 
new  drive  shaft  designed  to  improve  safety  (1986:389).  In  1990,  the  H-6-530  aircraft 
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began  a  no  tail  rotor  (Notar)  system  modification.  One  effect  of  removing  the  tail  rotor 
was  improved  safety  margins  due  to  weight  and  power  savings  (Brown,  1990:44-45). 

CRM  History.  As  stated  in  Chapter  II,  research  regarding  the 
human  factor  program  of  crew  resource  management  (CRM)  has  been  ongoing  since  the 
1970s.  The  Army,  recognizing  the  need  to  address  crew  errors,  first  introduced  a  6-8 
hour  block  of  CRM-type  training  to  helicopter  pilot  candidates  in  1983.  In  1984,  the 
training  was  revised  to  a  two  hour  block  and  retitled  Dynamics  of  Aircrew  Coordination 
Training.  In  1987,  the  Vice  Chief  of  Staff  of  the  Army  directed  the  Army  Research 
Institute  (ARI)  to  initiate  a  research  and  development  program  to  reduce  accident  rates 
through  various  means  including  training.  ARTs  subsequent  research  of  aviation 
accidents  showed  that  the  majority  of  human  error-related  accidents  involved  crew 
coordination  errors  (Simon,  1992:1-1).  In  1988,  the  Army  Safety  Center  (ASC) 
incorporated  aircrew  coordination  training  (ACT)  into  its  Aviation  Safety  Officer  Course. 
From  1990  to  1992  the  ARI,  the  Army  Aviation  Center,  and  the  Dynamics  Research 
Corporation  coordinated  their  efforts  to  conduct  aircrew  coordination  studies  and  revise 
aircrew  training  manuals.  The  research  culminated  in  aircrew  coordination  training  and 
evaluation  materials  exportable  to  the  field.  ACT  was  officially  implemented  Army-wide 
in  1994  (Directorate,  1998).  The  “central  feature  of  the  Army's  crew  coordination 
training  is  that  it  is  designed  to  reduce  the  number  of  accidents  (Simon,  1995:45). 

These  historical  events  could  confound  any  impact  RM  may  have  had  on  reducing 
the  mishap  rates.  The  inability  to  quantify  these  major  contributors  to  mishap  prevention 
interferes  with  quantifying  the  effectiveness  of  risk  management.  However,  there  may  be 
a  correlation  between  the  presence  of  RM  implementation  and  the  aviation  mishap  rate. 
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Maturation.  Maturation  concerns  the  effect  time  has  on  the  subjects  of  the 


experiment.  Since  this  research  covers  a  relatively  long  period,  those  whose  performance 
can  affect  the  mishap  rate  may  have  matured  over  that  time.  This  maturation  may  be  in 
the  form  of  aviation  experience,  training,  and  using  safer  equipment  as  described  in 
Chapter  II.  Concerning  maturation  after  RM  implementation,  it  is  assumed  that  the  more 
Army  aviators  practice  RM,  the  more  likely  the  mishap  rate  would  decrease.  In  this 
sense,  the  threat  of  maturation  is  similar  to  the  threat  of  history. 

Testing.  If  subjects  are  given  a  test,  it  is  likely  that  a  learning  effect  will 
occur  that  will  influence  scores  of  a  second  test  on  the  same  material.  For  this  research,  it 
was  not  possible  to  give  a  series  of  tests  in  this  manner.  Therefore,  the  testing  threat  to 
internal  validity  is  low. 

Instrumentation.  Instrumentation  concerns  the  way  measurements  are 
taken  to  obtain  results.  If  changes  are  made  in  instrumentation,  it  is  likely  that  the  results 
will  not  be  consistent.  In  this  research,  the  way  the  mishap  rate  was  calculated  did 
change  in  1984.  This  issue  will  be  addressed  in  the  methodology  section. 

Selection.  The  selection  of  subjects  for  control  and  experimental  groups  is 
important  in  that  they  need  to  be  equivalent  to  minimize  the  threat  to  internal  validity.  In 
this  research,  the  Army  is  the  experimental  group  and  the  Air  Force  is  the  control  group. 

It  is  true  that  their  aviation  components  are  not  equivalent  in  every  respect  but,  as  was 
discussed  in  Chapter  II,  there  is  enough  similarity  concerning  RM/ORM  that  this  threat 
can  be  considered  minimal. 

Regression.  The  regression  factor  is  in  effect  when  groups  are  selected 
based  on  extremely  high  or  low  results.  When  tested  after  a  treatment,  the  extreme 
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results  move  toward  their  expected  value.  For  this  research,  regression  analysis  is 
performed.  As  a  result,  extreme  mishap  rates,  outliers,  are  evaluated  and  removed  as 
necessary.  The  internal  threat  of  regression  is  minimal 

Experimental  Mortality.  This  threat  concerns  the  situation  where,  as  an 
experiment  continues,  subjects  are  likely  to  drop  out,  thus  affecting  the  results.  In  this 
study,  it  is  assumed  that  attrition  is  not  a  problem  because  if  one  member  of  the  aviation 
community  leaves,  another  of  similar  quality  will  replace  him.  Hence,  this  threat  to 
internal  validity  is  minimal. 

Interaction.  The  effect  of  interaction  between  any  of  the  previous  seven 
threats  to  internal  validity  can  be  confounded  with  the  effect  of  a  treatment,  X.  In  this 
study,  maturation  and  history  have  the  potential  to  confound  the  impact  of  RM. 
However,  since  the  duration  of  the  experiment  is  relatively  long,  maturation  and  history 
could  be  considered  one  threat.  The  other  five  threats  are  not  likely  to  interact. 
Therefore,  the  threat  of  interaction  is  minimal. 

External  Validity.  External  validity  concerns  the  ability  to  generalize  results  to 
other  populations.  Both  Campbell  and  Emory  list  three  major  threats  to  the  external 
validity  of  an  experiment  where  X  represents  the  treatment  (in  this  case,  RM):  1) 
interaction  of  testing  and  X,  2)  interaction  of  selection  and  X,  and  3)  reactive 
arrangements. 

Interaction  of  Testing  and  RM.  This  threat  can  be  high  if  a  pretest 
occurred  before  X.  How  the  subjects  respond  to  X  because  of  previous  exposure  can 
confound  the  results.  In  this  research,  it  is  assumed  that  no  testing  or  formalized  practice 
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of  RM  occurred  in  Army  aviation  before  RM  implementation  occurred  in  1987. 
Therefore,  this  threat  is  considered  minimal  to  the  generalizability  of  the  research  results. 

Interaction  of  Selection  and  RM.  This  threat  concerns  how  subjects  are 
selected  from  a  population  for  a  test,  or  in  this  case  the  treatment  of  RM,  and  the 
generalizability  of  the  results.  For  this  research,  die  population  of  subjects  is  the  Army 
aviation  and  safety  community.  The  results,  then,  will  be  generalizable  to  the  Army. 
However,  as  presented  in  Chapter  I,  the  basic  research  question  of  this  thesis  is  to 
determine  if  the  Air  Force  can  expect  its  mishap  rate  to  significantly  decline  after  ORM 
implementation.  It  is  not  the  purpose  to  directly  compare  Army  and  Air  Force  mishap 
rates  but  to  predict  the  effect  ORM  may  have  on  the  Air  Force  aviation  mishap  rate  based 
on  the  effect  RM  may  have  had  on  the  Army  aviation  mishap  rate.  It  is,  then,  the 
intention  to  be  able  to  generalize  the  results  to  the  Air  Force.  This  generalization  can  not 
be  based  on  statistical  results  alone  but  in  conjunction  with  those  investigative  questions 
addressed  in  Chapter  II.  Therefore,  the  threat  of  interaction  of  selection  and  RM  is 
considered  minimal  to  the  generalizability  of  the  research  results. 

Reactive  Arrangements.  This  threat  concerns  the  effect  that  the 
environment  in  which  testing  occurs  may  have  on  external  validity.  If  testing  is 
performed  in  an  artificial  environment  separate  from  it  original  population,  one  may  not 
be  able  to  validly  generalize  to  that  population.  In  this  research,  neither  the  Army  nor  Air 
Force  flying  environments  are  artificial  settings  in  which  to  test  RM/ORM.  The  Army 
flying  environment  may  be  significantly  different  from  that  of  the  Air  Force.  However, 
since  RM/ORM  is  a  decision-making  process  applicable  to  any  flying  situation,  it  is 
assumed  that  those  in  the  aviation  community  apply  RM/ORM  appropriately  to  their 
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situation.  Therefore,  the  threat  of  reactive  arrangement  is  considered  minimal  to  the 
generalizability  of  the  research  results. 

Summary.  This  section  addressed  the  threats  to  both  internal  and  external  validity 
to  the  research.  The  major  threat  to  its  internal  validity  is  history.  Various  historical 
events  associated  with  aviation  mishap  prevention  and  contemporaneous  with  RM  were 
presented  which  could  confound  the  effect  RM  may  have  had  on  the  mishap  rate.  Since 
the  manner  in  which  the  mishap  rate  was  calculated  changed  in  1984,  instrumentation 
could  also  influence  the  effect  RM  may  have  had  on  the  mishap  rate. 

Methodology,  Results,  and  Analysis 

Overview.  Chapter  I  presented  five  investigative  questions.  The  first  two  were 
answered  in  Chapter  II.  The  remaining  three  are  addressed  in  this  chapter  using 
statistical  analysis.  The  questions  are: 

1)  Was  there  a  significant  difference  in  the  Army  aircraft  mishap  rates  after  the 
implementation  of  RM?  (Investigative  Question  3). 

2)  If  there  was  a  significant  difference,  was  the  change  due  to  the  implementation 
of  RM?  (Investigative  Question  3a). 

3)  If  there  was  a  significant  difference  in  the  mishap  rate,  how  much  of  an  effect 
was  the  implementation  of  RM?  (Investigative  Question  3b). 

For  each  question,  the  methodology  is  first  presented  followed  by  the  results  and 
an  analysis  of  the  results.  All  statistical  tests  were  performed  using  JMP®  Version  3  and 
Mathcad®  7.  The  Army  data  analyzed  is  found  in  Appendix  A  and  presented  graphically 
in  Figures  3  and  4.  The  1991  data  point  was  removed  from  the  analysis  because  the 
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relatively  high  1991  Class  A  mishap  rate  can  be  attributed  to  the  preparation  for  and 
execution  of  Operation  Desert  Shield/Storm.  Additionally,  through  statistical  analysis, 
the  data  point  was  found  to  be  an  outlier  for  the  Class  A  mishap  rate  (Appendix  C). 


Figure  3.  Army  Class  A  Aviation  Mishap  Rate 


Figure  4.  Army  Class  B-C  Aviation  Mishap  Rate 


35 


After  the  1991  data  point  was  removed,  the  remaining  data  was  shown  to  be  from  a 
normal  distribution  (Appendix  D).  The  data  point  was  not  an  outlier  in  the  B-C  category 
(Appendix  E);  however,  it  was  removed  to  keep  the  analysis  consistent. 

Investigative  Question  3.  Was  there  a  significant  difference  in  the  Army  aircraft 
mishap  rates  after  the  implementation  of  RM?  To  address  this  question,  two  statistical 
tests  were  performed:  1)  comparison  of  means  test,  and  2)  a  comparison  of  variances. 

Comparison  of  Means. 

Methodology.  Just  looking  at  Figures  3  and  4,  one  can  intuitively 
say  that  the  rates  after  1987  are  lower  than  those  before  1987.  However,  to  answer  this 
question  statistically,  a  comparison  of  means  test  for  small  samples  was  considered 
according  to  McClave  (372-378).  The  means  compared  would  be  for  the  periods  1973- 
1987  (before  RM)  and  1988-1998  (after  RM)  for  both  Class  A  and  Class  B-C  mishap 
rates.  Three  assumptions  must  be  met  for  this  test  to  be  valid.  The  first  assumption  is 
that  both  sampled  populations  must  have  relative  frequency  distributions  that  are 
approximately  normal.  This  assumption  is  satisfied  through  an  analysis  of  the  residuals 
(Appendix  D  and  E).  The  second  assumption  is  that  the  samples  are  randomly  and 
independently  selected  from  the  population.  To  satisfy  this  assumption,  both  samples 
would  be  randomly  and  independently  selected  from  the  population  in  that  all  data  points 
in  the  population  except  the  outlier  could  be  used  in  the  analysis.  The  third  assumption  is 
that  the  population  variances  are  equal.  To  attempt  to  satisfy  this  assumption,  an  analysis 
of  the  population  variances  of  the  values  could  be  performed.  However,  this  approach 
cannot  satisfy  this  assumption  because  this  research  is  dealing  with  time  series  data.  As  a 
result,  the  direct  comparison  of  means  test  is  not  appropriate. 
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The  problem  with  performing  a  direct  comparison  of  means  is  that  when  time 
series  data  is  analyzed,  trend  components  must  be  removed  before  any  mean  analysis  can 
be  performed.  If  the  time  series  is  stationary,  the  values  will  “fluctuate  with  constant 
variation  around  a  constant  mean”  (Bowerman,  1987:26).  Since  time  series  data  is 
considered  in  this  research  and  the  mishap  rate  processes  do  not  appear  to  be  stationary, 
the  direct  comparison  of  means  is  an  inappropriate  test  for  this  data.  A  transformation  of 
the  data  to  stationarity,  however,  does  permit  a  comparison  of  means. 

Since  the  mishap  rate  is  expressed  as  a  percentage  (as  defined  in  Chapter  II),  a 
way  to  transform  the  data  into  a  form  that  is  suitable  for  comparison  of  means  is  to 
compute  the  percentage  period  index  (PPI)  using  the  procedure  described  in  Makridakis 
(1983:171).  PPI  is  a  period-to-period  (in  this  case  year-to-year)  percentage  change.  A 
difference  of  means  test  comparing  the  mean  PPI  for  each  period  is  then  possible.  If  the 
means  are  different,  it  indicates  that  a  process  change  has  occurred. 

To  run  the  test,  the  PPI  of  the  first  value  in  the  data  set  is  set  equal  to  a  constant  to 
provide  an  order  of  magnitude.  Each  PPI  thereafter  is  determined  by  computing  the  ratio 
of  the  current  rate  to  the  previous  rate  and  multiplying  the  ratio  by  the  constant.  In 
general: 

PPI  =  [(Ratej+i )  /  (Ratei )]  x  C  from  i  =  1  to  n  (1) 
where  C  =  10.  The  hypotheses  for  the  test  were: 

Ho:  the  means  are  equal 
Ha:  the  means  are  not  equal 

To  reject  the  null  hypothesis  in  the  comparison  of  means  t-test,  the  t-statistic  must  be 
greater  than  the  t-critical  value.  The  level  of  significance  for  the  t-test  was  a  =  .05. 
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Results.  Table  3  summarizes  the  results  of  the  PPI  transformation 


that  are  presented  in  Appendix  F. 


Table  3.  Percentage  Period  Index  Results 


Class 

Year 

Mean  PPI 

t-crit 

t-stat 

Reject  Null? 

A 

1973-1987 

9.73 

1.714 

.198 

No 

1988-1998 

9.92 

B-C 

rj973-1987 

9.32 

1.714 

1.601 

No 

1988-1998 

11.02 

The  results  showed  that  there  is  enough  evidence  to  suggest  that  the  mean  PPIs 
for  both  Class  A  and  Class  B-C  mishap  rate  categories  were  not  significantly  different  at 
the  a  =  .05  level  of  significance.  Figures  5  and  6  present  the  PPIs  graphically  for  both 
Class  A  and  Class  B-C  categories  respectively. 
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Figure  5.  Class  A  Period  Percentage  Index 

Analysis.  Since  there  was  no  significant  difference  in  the  mean  PPIs 
for  both  categories,  the  mean  percentage  change  of  the  mishap  rate  from  year  to  year  can 
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be  considered  the  same  for  both  periods.  This  indicates  that  the  process  to  reduce  the 
mishap  rate  has  not  changed  since  1987. 


Figure  6.  Class  B-C  Period  Percentage  Index 


Comparison  of  Variances. 


Methodology.  As  discussed  earlier,  it  is  not  appropriate  to 


compare  the  variances  of  trended  data.  However,  it  is  appropriate  and  useful  to  compare 
the  variances  of  the  residuals  of  the  mishap  rate  values  regressed  against  the  fiscal  year. 


Since  the  mean  of  the  residuals  is  zero,  a  comparison  between  the  variances  of  the 


residuals  is  possible.  A  change  in  variance  of  the  population  of  the  residuals  from  one 
period  to  the  next  will  at  least  determine  if  a  process  change  occurred.  In  looking  at 
Figures  3  and  4,  it  would  be  reasonable  to  expect  the  variance  of  the  residuals  for  the 
Class  B-C  mishap  rate  to  be  different  as  clearly  a  process  change  has  occurred  around 
1987.  A  difference  in  variance  is  not  as  apparent  for  the  Class  A  category. 
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A  comparison  of  the  variances  of  the  residuals  of  the  mishap  rate  values  for  the 
time  periods  1973-1987  (before  RM)  and  1988-1998  (after  RM)  for  both  Class  A  and 
Class  B-C  mishap  categories  was  performed  as  described  in  McClave  (1998:408-415). 
The  F-statistic  is  computed  by  placing  the  larger  variance  as  the  numerator  and  the 
smaller  as  the  denominator.  The  critical  F  value  is  determined  based  on  the  numerator 
and  denominator  degrees  of  freedom.  If  the  residual  variances  are  the  same,  one  can 
conclude  that  there  has  been  no  process  change.  The  hypotheses  for  the  test  were: 

Ho:  The  residual  variances  are  equal 
Ha:  The  residual  variances  are  not  equal 

If  the  F-statistic  is  greater  than  the  critical  F  value,  then  the  F-statistic  falls  in  the 
rejection  region  and  the  null  hypothesis  is  rejected.  This  concept  is  illustrated  in  Figure 
7. 


Figure  7.  F-Distribution  (McClave,  1998:410) 

Results.  The  results  of  the  comparison  of  variance  test  are 
presented  in  Appendix  G  and  summarized  in  Table  4.  Since  the  F-statistic  for  the  Class 
A  mishap  rate  did  not  fall  in  the  rejection  region,  the  null  hypothesis  is  not  rejected.  The 
residual  variances  are  equal.  Since  the  F-statistic  for  the  Class  B-C  mishap  rate  did  fall 
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Table  4.  Variance  Test  Results 


Class 

Year 

Variance 

F-crit 

F-stat 

Reject  Null? 

H 

1973-1987 

.115 

3.803 

1.435 

No 

1988-1998 

.080 

B-C 

1973-1987 

10.529 

3.803 

12.983 

Yes 

1988-1998 

.811 

in  the  rejection  region,  the  null  hypothesis  is  rejected.  The  variances  are  not  equal. 

Analysis.  Since  the  residual  variances  for  the  Class  A  mishap  rate 
category  are  equal,  there  is  not  enough  evidence  to  conclude  that  a  process  change  has 
occurred.  However,  for  the  Class  B-C  category,  since  the  residual  variances  were  not 
equal,  there  is  enough  evidence  to  conclude  that  a  process  change  has  occurred.  Since 
RM  was  instituted  in  1987,  this  is  the  result  expected  from  having  reviewed  Figure  4. 

Summary.  Investigative  question  3  asked  whether  there  was  significant 
difference  in  the  mishap  rate  after  the  implementation  of  RM.  A  difference  of  means  test 
was  performed  on  the  percentage  period  indexes  for  both  mishap  rate  categories.  The 
results  showed  no  significant  difference  in  the  mean  PPI  for  either  category.  This 
indicated  that  no  process  change  had  occurred.  A  comparison  of  the  variances  of  the 
residuals  of  the  mishap  rates  was  then  performed  to  confirm  the  absence  of  a  process 
change.  The  results  confirmed  that  for  the  Class  A  mishap  category,  no  process  change 
was  evident.  However,  for  the  Class  B-C  mishap  category,  a  process  change  was  evident. 
A  more  powerful  method  of  determining  whether  there  was  significant  change  in  the 
mishap  rates  is  to  employ  discontinuous  piecewise  linear  regression.  The  procedure  will 
simultaneously  address  whether  the  process  change  as  found  in  the  Class  B-C  category 
was  due  to  the  implementation  of  RM. 
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Investigative  Question  3a.  If  there  was  a  significant  difference,  was  the  change 
due  to  the  implementation  of  RM? 

Methodology.  To  address  this  question,  discontinuous  piecewise  linear 
regression  was  performed  for  the  periods  1973-1998.  Discontinuous  piecewise  linear 
regression  is  used  to  determine  if  a  slope  and/or  intercept  change  has  occurred  at  a 
particular  break  point  or  points  (Neter,  1996:474-478).  A  model  with  two  variables  and  a 
break  point,  C,  would  look  like  this: 

E(MR)  =  p0  +  Pi*Xi  +  p2*(Xi-C)*X2  +  p3*X2  (2) 

where  p0  is  the  Y-axis  intercept,  pi  is  the  slope  of  the  regression  line  for  the  period  before 
treatment  C,  Pi+p2  is  the  slope  of  the  regression  line  for  the  period  after  C,  and  P3  is  the 
jump  in  intercept  at  C.  Figure  8  illustrates  this  concept. 


(Neter,  1996:478) 


If  there  is  no  significant  change  in  the  slope  of  the  regression  line  at  point  C,  then  one 
would  expect  p2  to  be  zero.  Similarly,  if  there  is  no  significant  change  in  the  intercept  at 
C,  one  would  expect  P3  to  be  zero. 
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Typically,  with  a  process  or  policy  change,  one  would  expect  to  see  a  significant 
change  in  the  data  results  if  the  policy  was  effective.  An  effective  treatment  would 
produce  a  shift  in  slope  and/or  intercept.  A  shift  in  intercept  with  no  change  in  slope  or  a 
change  in  slope  with  no  change  in  intercept  could  reveal  whether  the  treatment  caused  a 
significant  change  in  the  process  (Campbell,  1963:43). 

Since  Army  risk  management  began  in  1987,  one  might  expect  to  see  its  effect 
evidenced  by  a  downward  shift  in  the  intercept  and/or  an  increased  negative  slope  in  the 
mishap  rate  regression  line  after  1987. 

The  full  model  consisted  of  two  variables:  FY  and  RM.  FY  is  the  fiscal  year. 

RM  represents  the  presence  of  risk  management.  From  1973-1987  RM  =  0  and  from 
1988-1998  RM  =  1.  The  breakpoint,  C,  is  87.  The  full  model  was  as  follows: 

E(MR)  =  p0  +  Pi*FY  +  p2*(FY-87)*RM  +  p3*RM  (3) 

where  Po  is  the  Y-axis  intercept,  pi  is  the  slope  of  the  regression  line  for  the  period  1973- 
1987,  P1+P2  is  the  slope  of  the  regression  line  for  the  period  1988-1998,  and  P3  is  the 
jump  in  intercept  between  1987  and  1988.  As  a  result,  three  hypotheses  were  made  for 
each  mishap  category  (A,  and  B-C).  The  first  hypothesis  was: 

Ho:  Pi  =  p2  =  p3  =  0 
Ha:  The  P's  *  0 

The  significance  of  the  pi  and  P3  terms  can  be  directly  determined  by  their  p-values  from 
the  overall  F-test  results  of  the  full  model.  However,  a  partial  F-test  must  be  performed 
on  the  reduced  model  to  determine  the  significance  of  the  slope  of  the  second  regression 
line,  Pi  +  p2.  Asa  result,  the  second  hypothesis  was: 
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Ho:  Pi  +  p2  =  0 
Ha:  Pi  +  p2*0 

Additionally,  to  determine  if  the  slopes  of  the  two  regression  lines  are  significantly 
different  from  each  other,  a  second  partial  F-test  must  be  performed.  If  p2  is  zero,  then 
the  slope  of  the  second  line  will  not  be  significantly  different  from  the  slope  of  the  first 
line.  Hence,  the  third  hypothesis  was: 

Ho:  p2  =  0 
Ha:  p2  ^  0 

The  results  and  analysis  of  the  Class  A  mishap  category  are  first  presented  followed  by 
those  in  the  Class  B-C  mishap  category.  All  tests  were  performed  at  an  a  =  .05  level  of 
significance. 

Class  A  Mishap  Category- 

Results.  Tables  5  and  6  summarize  the  discontinuous  piecewise 
linear  regression  overall  and  partial  F-test  results  respectively  for  the  Class  A  mishap 
category.  With  the  given  p-values,  the  overall  F-test  indicates  that  1)  the  slope  of  the  first 
line,  Pi,  is  significantly  different  from  zero  and  2)  that  there  is  no  significant  jump  at 
1987.  The  partial  F-tests  on  the  reduced  model  indicate  that  1)  the  slope  of  the  second 
line,  Pi+p2,  is  significantly  different  from  zero,  and  2)  the  slopes  of  the  regression 


Table  5.  Overall  F-test  Results:  Class  A  Mishap  Category 


Term 

Beta 

Beta 

Coefficients 

P- Value 

Reject  Null? 

Intercept 

Po 

11.016 

.0001 

N/A 

FY 

Pi 

-.102 

.0001 

Yes 

(FY-87)xRM 

P2 

.011 

.7672 

N/A 

RM 

P3 

-.128 

.6525 

No 
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Table  6.  Partial  F-test  Results:  Class  A  Mishap  Category 


Beta 

Slope  of  Line 
1988-1998 

Reject  Null? 

Pl+02 

-.091 

4.325 

7.772 

Yes 

P2 

N/A 

4.325 

.09 

No 

lines  are  not  significantly  different  from  one  another.  Figure  9  shows  the  discontinuous 
piecewise  linear  regression  for  the  Class  A  mishap  rate.  The  graph  suggests  a  continuous 
downward  trend  in  the  Class  A  mishap  rate. 


Figure  9.  Class  A  Mishap  Rate  (Risk  Management) 


Analysis.  Since  there  was  no  shift  in  the  regression  line  at  1987 
and  the  slopes  of  the  two  regression  lines  are  not  significantly  different,  there  is  no 
indication  that  the  presence  of  RM  has  had  any  effect  on  the  Class  A  mishap  rate.  These 
results  confirm  the  finding  of  Investigative  Question  3  that  no  process  change  has 
appeared  to  have  occurred  in  the  Class  A  mishap  rate  category. 
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Class  B-C  Mishap  Category. 


Results.  Tables  7  and  8  summarize  the  discontinuous  piecewise 
linear  regression  overall  and  partial  F-test  results  respectively  for  the  Class  B-C  mishap 
category. 


Table  7.  Overall  F-test  Results:  Class  B-C  Mishap  Category 


Term 

Beta 

Beta 

Coefficients 

P- Value 

Reject  Null? 

Intercept 

Po 

101.495 

.0001 

Yes 

FY 

Pi 

-1.100 

.0001 

Yes 

(FY-87)xRM 

Pa 

1.541 

.0001 

N/A 

RM 

P3 

-2.060 

.3602 

No 

Table  8.  Partial  F-test  Results:  Class  B-C  Mishap  Category 


Beta 

Slope  of  Line 
1988-1998 

Fcrit 

Fstat 

Reject  Null? 

Pi+p2 

.441 

4.844 

1.573 

No 

Hu9H 

N/A 

4.325 

26.497 

Yes 

With  the  given  p-values,  the  overall  F-test  indicates  that  1)  the  slope  of  the  first  line,  pi, 
is  significantly  different  from  zero  and  2)  that  there  is  no  significant  jump  at  1987.  The 
partial  F-tests  on  the  reduced  model  indicate  that  1)  the  slope  of  the  second  line,  Pi+p2,  is 
not  significantly  different  from  zero,  and  2)  that  the  slopes  of  the  regression  lines  are 
significantly  different  from  one  another.  Figure  10  shows  the  discontinuous  piecewise 
linear  regression  for  the  Class  B-C  mishap  rate.  The  graph  suggests  an  upward  trend  in 
the  Class  B-C  mishap  rate  category  since  1987. 

Analysis.  In  the  comparison  of  variances  performed  earlier,  a 
process  change  was  confirmed  for  the  Class  B-C  category.  If  this  change  were  due  to 


46 


RM,  a  shift  in  the  regression  line  at  1987  and/or  an  increased  negative  slope  of  the  second 
regression  line  after  1987  would  indicate  that  RM  effected  this  change.  However,  since 


Figure  10.  Class  B-C  Mishap  Rate  (Risk  Management) 


there  was  no  shift  and  the  slope  of  the  regression  line  after  1987  actually  reversed  course 
and  became  positive,  it  can  be  said,  at  the  least,  that  RM  has  not  had  its  intended  effect. 
There  may  be  several  possibilities  for  this.  It  is  possible  that,  with  the  emphasis  on  safety 
that  RM  brings  with  it,  a  stricter  reporting  policy  of  Class  B  and  C  mishaps  may  have 
been  perceived  by  commanders.  It  is  also  possible  that  the  flight-related  confound 
analyzed  in  the  next  section  affected  mishap  reporting.  In  any  event,  the  mishap  cause 
factors  discussed  in  Chapter  II  outweighed  the  mishap  prevention  factors  for  reasons 
beyond  the  scope  of  this  research. 

Summary.  Investigative  Question  3a  asked  whether  any  significant  difference 
found  in  Investigative  Question  3  was  due  to  RM.  Discontinuous  piecewise  linear 
regression  analysis  was  employed  to  address  this  question.  For  the  Class  A  mishap 
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category,  the  test  reinforced  the  previous  finding  that  no  process  change  occurred.  The 
conclusion  is  that  RM  has  not  affected  the  Class  A  mishap  rate.  For  the  Class  B-C 
mishap  category,  the  test  confirmed  a  process  change  identified  in  previous  analysis.  In 
fact,  since  1987,  the  mishap  rates  have  increased.  Several  possibilities  for  this  occurrence 
were  discussed. 

Investigative  Question  3b.  If  there  was  a  significant  difference  in  the  mishap  rate, 
how  much  of  an  effect  was  the  implementation  of  RM? 

Although  the  test  performed  for  Investigative  Question  3a  found  a  potential 
significant  difference  (change  in  slope)  in  the  Class  B-C  mishap  rate  since  RM 
implementation,  it  does  not  appear  that  RM  has  had  its  intended  effect.  This  may  be 
influenced  by  the  confounding  effect  of  flight-related  mishaps  in  the  rate  calculation 
discovered  during  data  collection.  How  much  of  an  effect  this  confound  has  had  on  the 
mishap  rate  may  change  the  determination  of  how  much  of  an  effect  RM  has  had  on  the 
mishap  rate. 

Methodology.  The  previous  analysis  for  the  effect  of  RM  after  1987 
ignored  the  fact  that  flight-related  data  were  included  in  the  mishap  rate  calculation  for 
the  period  1973-1983.  In  order  to  test  the  impact  of  the  confounding  effect  this  may  have 
had  on  the  rate  calculation,  discontinuous  piecewise  linear  regression  analysis  was 
performed  for  the  period  1973-1987  for  both  mishap  categories.  The  breakpoint,  C,  was 
84.  The  two  variables  for  test  one  were  FY  and  FR.  FR  represented  the  presence  of 
flight-related  mishaps  included  in  the  mishap  rate  calculation.  The  full  model  of  the 
expected  value  of  the  mishap  rate  was  as  follows: 

E(MR)  =  p0  +  pi*FY  +  P2*(FY-C)*FR  +  p3*FR  (4) 
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where  po  is  the  Y-axis  intercept.  Pi  is  the  slope  of  the  regression  line  for  the  first  period 
regression  line,  P1+P2  is  the  slope  of  the  second  period  regression  line,  and  p3  is  the  jump 
in  intercept  between  periods.  Similar  testing  was  performed  for  the  period  1984-1998 
(flight  mishap  only  data)  to  see  if  there  was  any  change  from  the  previous  RM  test  in  the 
effect  of  RM  after  1987.  The  breakpoint,  C,  was  87.  The  two  variables  for  test  two  were 
F  Y  and  RM  just  as  in  the  full  model  presented  in  the  previous  RM  piecewise  analysis.  A 
summary  of  the  variables  for  both  tests  for  the  Class  A  and  Class  B-C  mishap  categories 
is  presented  in  Table  9. 

Table  9.  Variables  for  Piecewise  Regression  Tests 


Test# 

Period 

FR 

RM 

1 

1973-1983 

0 

N/A 

1984-1987 

1 

N/A 

2 

1984-1987 

N/A 

0 

1988-1998 

N/A 

1 

The  same  hypotheses  as  in  the  previous  analysis  apply  here  as  well.  The  results  and 
analysis  for  the  Class  A  category  are  first  presented  followed  by  those  of  the  Class  B-C 
category. 

Class  A  Mishap  Category. 

Results:  Test  One.  Tables  10  and  1 1  summarize  the  overall  and 
partial  F-test  results  respectively  from  test  one  (1973-1987)  of  the  Class  A  mishap 
category.  With  the  given  p-values,  the  overall  F-test  indicates  that  1)  the  slope  of  the  first 
line,  pi,  is  significantly  different  from  zero  and  2)  that  there  is  no  significant  jump  at 
1983.  The  partial  F-tests  on  the  reduced  model  indicate  that  1)  the  slope  of  the  second 


49 


line,Pi+p2,  is  not  significantly  different  from  zero,  and  2)  that  the  slopes  of  the  regression 
lines  are  not  significantly  different  from  one  another.  Figure  1 1  presents  the  results 
graphically. 


Table  10.  Overall  F-test  Results:  Class  A,  Test  One,  1973-1987 


Term 

Beta 

Beta 

Coefficients 

P-Value 

Reject  Null? 

Intercept 

Po 

12.524 

.0008 

N/A 

FY 

Pi 

-.122 

.0051 

Yes 

(FY-84)  x  FR 

P2 

-.083 

.6274 

N/A 

FR 

P3 

.479 

.3519 

No 

Table  11.  Partial  F-test  Results:  Class  A,  Test  One,  1973-1987 


Beta 

Slope  of  Line 
1984-1987 

m 

Fstal 

Reject  Null? 

P1+P2 

-.205 

4.844 

1.573 

No 

P2 

N/A 

4.844 

.249 

No 

Analysis:  Test  One.  The  partial  F-tests  showed  that  the  slope  of 
the  regression  line  for  the  1973-1984  period  was  significant,  that  the  slope  the  regression 
line  from  1984-1987  was  not  significant  and  yet  the  slope  of  the  two  lines  were  not 
significantly  different  from  one  another.  The  insignificant  slope  of  the  second  regression 
line  is  most  likely  due  to  the  high  variability  associated  with  having  only  four  data  points. 
There  is  not  much  statistical  power  with  only  four  data  points.  However,  a  look  at  Figure 
1 1  confirms  that  the  slopes  of  the  two  lines  are  not  much  different.  To  draw  conclusions 
from  this  test,  more  weight  was  put  on  the  result  that  the  slopes  of  the  two  lines  were  not 
significantly  different.  Therefore,  test  one  indicated  that  there  was  no  significant  change 
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in  the  Class  A  mishap  rate  due  to  the  inclusion  of  flight-related  mishaps  in  the  rate 
calculation. 

Results:  Test  Two.  The  overall  and  partial  F-test  results 
respectively  from  test  two  (1984-1998)  are  presented  in  Tables  12  and  13.  With  the 


Figure  11.  Class  A  Mishap  Rate  (Flight-Related  Confound) 


given  p-values,  the  overall  F-test  indicates  that  1)  the  slope  of  the  first  line,  Pi,  is  not 
significantly  different  from  zero  and  2)  that  there  is  no  significant  jump  at  1987. 


Table  12.  Overall  F-test  Results:  Class  A,  Test  Two,  1984-1998 


Term 

Beta 

Coefficients 

P-Value 

Reject  Null? 

Intercept 

Po 

19.93 

.1401 

N/A 

FY 

Pi 

-.205 

.1890 

No 

(FY-87)xRM 

P2 

.115 

.4590 

N/A 

RM 

p3 

-.063 

.8618 

No 
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Table  13.  Partial  F-test  Results:  Class  A,  Test  Two,  1984-1998 


Beta 

Slope  of  Line 
1988-1998 

IEB 

m 

Reject  Null? 

P1+P2 

4.965 

8.158 

Yes 

P2 

N/A 

4.965 

.593 

No 

The  partial  F-tests  on  the  reduced  model  indicate  that  1)  the  slope  of  the  second  line, 
P1+P2,  is  significantly  different  from  zero,  and  2)  that  the  slopes  of  the  regression  lines 
are  not  significantly  different  from  one  another.  Figure  1 1  depicts  the  results  of  test  two 
graphically. 

Analysis:  Test  Two.  Test  two  had  similar  results  to  test  one  for 
the  1984-1987  period.  Again,  the  high  variability  of  having  only  four  data  points  of 
flight-only  mishaps  before  the  implementation  of  RM  probably  negates  any  relevant 
slope  that  likely  exists.  Conclusions  drawn  were  based  more  on  the  results  that  there  was 
no  jump  at  1987  and  that  the  slopes  of  the  two  lines  were  not  significantly  different  from 
one  another.  Therefore,  test  two  validated  that  the  flight-related  mishap  confound  does 
not  appear  to  change  the  fact  that  RM  has  not  affected  a  change  in  the  Class  A  mishap 
rate. 

Class  B-C  Mishap  Category. 

Results:  Test  One.  The  results  of  test  one  (1973-1987)  are 
summarized  in  Tables  14  and  15.  With  the  given  p-values,  the  overall  F-test  indicates 
that  1)  the  slope  of  the  first  line.  Pi,  is  significantly  different  from  zero  and  2)  that  there  is 
no  significant  jump  at  1987  at  the  a  =  .05  level  of  significance.  The  partial  F-tests  on  the 
reduced  model  indicate  that  1)  the  slope  of  the  second  line,  P1+P2,  is  not  significantly 
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Table  14.  Overall  F-test  Results:  Class  B-C,  Test  One,  1973-1987 


Term 

Beta 

Beta 

Coefficients 

P- Value 

Reject  Null? 

Intercept 

Po 

.0107 

N/A 

FY 

Pi 

-.622 

.0423 

Yes 

FR'xFR 

P2 

.435 

.7442 

N/A 

FR 

Pa 

-7.277 

.0839 

No 

Table  15.  Partial  F-test  Results:  Class  B-C,  Test  Two,  1973-1987 


Beta 

Slope  of  Line 
1984-1987 

Reject  Null? 

P1+P2 

-.187 

4.844 

.022 

No 

P2 

N/A 

4.844 

.112 

No 

different  from  zero,  and  2)  that  the  slopes  of  the  regression  lines  are  not  significantly 
different  from  one  another.  Figure  12  presents  the  results  graphically. 

Analysis:  Test  One.  Test  one  showed  the  jump  at  1983  to  be 
insignificant  at  a  =  .05  with  a  p-value  of  the  p3  term  of  .0839.  However,  by  looking  at 
Figure  12,  it  is  apparent  that  there  ought  to  be  a  significant  jump  at  1983.  Relaxing  the 
level  of  significance  to  a  =  .1  reveals  that  the  jump  is  statistically  significant.  Practically 
speaking,  this  makes  sense.  Flight-related  mishaps,  by  definition,  are  not  usually  going 
to  result  in  high  dollar  value  damage.  Therefore  they  tend  to  fall  into  the  Class  B-C 
mishap  category.  By  removing  the  Class  B-C  flight-related  mishaps  from  the  mishap  rate 
calculation  before  1983,  one  would  expect  a  significant  drop  in  the  mishap  rate.  This 
may  partially  explain  why  the  rate  appears  to  have  increased  since  1 987.  Based  on  the 
results  and  practical  judgement,  it  appears  that  the  flight-related  confound  has  had  a 
significant  effect  on  the  Class  B-C  mishap  rate.  Test  two  determined  how  this  change 
might  have  affected  any  influence  that  RM  may  have  had. 
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Results:  Test  Two.  The  results  of  test  two  (1984-1998)  are 


presented  in  Tables  16  and  17.  With  the  given  p- values,  the  overall  F-test  indicates  that 
1)  the  slope  of  the  first  line,  Pi,  is  not  significantly  different  from  zero,  and  2)  that  there  is 
no  significant  jump  at  1987.  The  partial  F-tests  on  the  reduced  model  indicate  that  1)  the 


Table  16.  Overall  F-test  Results:  Class  B-C,  Test  Two,  1984-1998 


Term 

Beta 

Beta 

Coefficients 

P- Value 

Reject  Null? 

Intercept 

Po 

21.511 

.6235 

N/A 

FY 

Pi 

-.187 

.7145 

No 

(FY-87)xRM 

P2 

.576 

.2822 

N/A 

RM 

Pa 

-.956 

.4340 

No 

Table  17.  Partial  F-test  Results:  Class  B-C,  Test  Two,  1984-1998 


Beta 

Slope  of  Line 
1988-1998 

Fcrit 

Fstal 

Reject  Null? 

P1+P2 

.389 

4.844 

13.41 

Yes 

N/A 

4.844 

Pi  .279 

No 

slope  of  the  second  line,  P1+P2,  is  significantly  different  from  zero,  and  2)  that  the  slopes 
of  the  regression  lines  are  not  significantly  different  from  one  another. 

Analysis:  Test  Two.  The  fact  that  the  slope  of  the  regression  line 
of  the  1984-1987  time  period,  pi,  is  not  significantly  different  from  zero  is  not  surprising 
because  that  was  the  same  result  from  test  one  where  the  slope  of  the  same  line  was 
represented  by  P1+P2.  Based  on  the  results,  the  apparent  effect  of  the  flight-related 
confound  did  not  affect  any  influence  that  RM  may  have  had. 
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Figure  12.  Class  B-C  Aviation  Mishap  Rate  (Flight-Related  Confound) 

Summary.  Given  a  significant  difference  in  the  mishap  rate  after  RM 
implementation.  Investigative  Question  3a  asked  how  much  of  an  effect  RM 
implementation  might  have  had  on  the  mishap  rate.  A  confound  discovered  during  data 
collection  was  that  before  1983,  the  mishap  rate  calculation  included  flight-related 
mishaps.  Therefore,  before  directly  focusing  the  investigative  question,  the  effect  of  the 
confound  had  to  be  addressed.  To  address  the  investigative  question,  then,  required  two 
steps.  First  analyses  of  the  data  in  the  period  1973-1 987  were  performed  with  a  break 
point  at  1 983.  This  compared  mishap  rates  including  flight-related  mishaps  (1973-1983) 
to  mishap  rates  based  on  flight  mishaps  only  but  excluding  the  years  after  RM 
implementation  (1984-1987).  Second,  analyses  of  the  data  in  the  period  1984-1998  were 
performed  with  at  break  point  at  1987.  This  compared  mishap  rates  based  on  flight 
mishaps  only  before  and  after  the  implementation  of  RM.  For  the  Class  A  mishap 
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category,  test  one  showed  no  significant  change  in  the  Class  A  mishap  rate  by  excluding 
flight-related  mishaps  from  the  rate  calculation  after  1983.  Test  two  validated  that 
accounting  for  the  flight-related  mishap  confound  did  not  change  the  earlier  conclusion 
that  RM  has  not  affected  a  change  in  the  Class  A  mishap  rate.  For  the  Class  B-C 
category,  test  one  showed  a  significant  change  (at  a  =  .1)  in  the  mishap  rate  by  excluding 
flight-related  mishaps  from  the  rate  calculation  after  1983.  However,  this  finding  did  not 
affect  any  influence  that  RM  may  have  had  on  the  mishap  rate. 

To  further  to  address  Investigative  Question  3b  and  the  flight-related  confound, 
stepwise  regression  analysis  was  performed. 

Stepwise  Regression  Analysis.  Stepwise  regression  is  normally  used  as  a 
screening  procedure  to  pare  down  a  large  number  of  independent  variables  to  the 
important  few  for  manageable  model  building  (McClave  1998:643-648).  Although  only 
three  variables  were  considered  in  the  previous  analyses  (FY,  RM,  and  FR),  stepwise 
regression  analysis  was  performed  as  an  additional  test  to  determine  the  relative 
importance  of  these  variables.  The  variables  that  are  selected  by  the  stepwise  process  are 
the  ones  that  significantly  contribute  to  the  model.  The  relatively  less  important  variables 
will  not  be  selected. 

Inputting  one  variable  at  a  time  into  the  model  performs  forward  stepwise 
regression.  As  each  new  variable  is  introduced,  t-tests  are  performed  to  determine 
whether  the  variable  belongs  in  the  model.  Backward  stepwise  regression  is  performed 
by  inputting  all  variables  into  the  model  at  one  time  and  performing  t-tests  on  each 
variable  to  determine  which  ones  to  extract  from  the  model. 
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Methodology.  Backward  stepwise  regression  was  conducted  using  JMP® 
Version  3  for  fiscal  year  periods  1973-1998  and  1984-1998  for  Class  A  and  Class  B-C 
mishap  categories.  To  represent  the  effect  of  the  mishap  reporting  method  (flight  and 
flight-related  mishaps  vs.  flight  mishaps  only),  FR  =  1  for  years  1973-1983  and  FR  =  0 
foryears  1984-1998.  Similarly,  to  represent  the  effect  of  the  presence  of  risk 
management,  RM  =  0  for  years  1973-1987  and  RM  —  1  for  years  1988-1998.  The  1984- 
1998  data  set  was  tested  for  both  mishap  rate  categories  to  see  if  the  effect  of  RM  could 
be  more  clearly  seen  without  the  effect  of  the  flight-related  variable.  Both  the  full  and 
reduced  mathematical  models  are  presented  in  Appendix  H.  Coefficients  of 
determination  (R2)  values  of  the  resulting  models  are  also  presented.  The  R2  value  is  the 
percent  of  variability  in  the  expected  mishap  rate  that  can  be  explained  by  using  the 
independent  variables  to  predict  the  expected  mishap  rate  in  the  straight  line  model 
(McClave,  1998:467).  The  higher  the  R2  value  the  more  variability  that  is  explained. 

The  Class  A  mishap  category  results  and  analysis  are  first  presented  followed  by  those  of 
the  Class  B-C  mishap  category. 

Class  A  Mishaps  1973-1998. 

Results  and  Analysis.  The  beta  coefficients  and  their 
corresponding  p-values  for  both  the  full  and  backward  stepwise  regression  results  are 
presented  in  Tables  18  and  19  respectively.  The  backward  stepwise  regression  analysis 
results  produced  an  R2  =  .8618.  It  also  showed  that  the  FY  term  was  the  only  term  that 
had  any  significant  effect  on  the  expected  mishap  rate.  This  model  indicates  that  the 
expected  mishap  rate  is  on  a  downward  trend  and  depends  mainly  upon  the  fiscal  year. 
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Table  18.  Stepwise  Full  Model:  Class  A,  1973-1998 


Terms 

Beta 

Beta  Coefficients 

P-Values 

Intercept 

Po 

19.93 

.1413 

FY 

Pi 

-.205 

.1929 

RM 

P2 

-10.0347 

.4613 

FR 

Pa 

-7.405818 

.5821 

FY*RM 

P12 

.1146212 

■MEEfeMi 

FY*FR 

Pl3 

.0834545 

.5972 

Table  19.  Stepwise  Results:  Class  A,  1973-1998 


Terms 

Beta 

Beta  Estimates 

P-Values 

Intercept 

Po 

11.190 

.0001 

FY 

Pi 

-.104 

.0001 

The  results  are  consistent  with  those  found  in  previous  regression  analyses.  It  does 
not  appear  that  the  inclusion  of  flight-related  mishaps  in  the  rate  calculation  was  a 
significant  factor.  Nor  does  it  appear  that  the  presence  of  RM  was  a  factor  in  the 
reduction  of  the  Class  A  mishap  rate. 

Class  A  Mishaps  1984-1998. 

Results  and  Analysis.  The  beta  coefficients  and  their 
corresponding  p- values  for  both  the  full  and  backward  stepwise  regression  results  are 
presented  in  Tables  20  and  21. 


Table  20.  Stepwise  Full  Model:  Class  A,  1984-1998 


Terms 

Beta 

Beta  Coefficients 

P-Values 

Intercept 

Po 

19.930 

.1401 

FY 

Pi 

-.205 

.1890 

RM 

P2 

-10.035 

.4505 

FY*RM 

P12 

.115 

.4590 
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Table  21.  Stepwise  Results:  Class  A,  1984-1998 


Terms 

Beta 

Beta  Coefficients 

P-Values 

Intercept 

Po 

11.826 

.0001 

FY 

Pi 

-.111 

.0001 

The  backward  stepwise  regression  analysis  results  produced  an  R  =  .7474.  It  also 
showed  that  the  FY  term  was  the  only  term  that  had  any  significant  effect  on  the  expected 
mishap  rate.  This  model  is  virtually  the  same  as  for  as  the  model  for  the  1973-1998 
period.  The  effect  of  RM  is  no  more  clearly  seen  in  this  test  than  in  the  previous  tests. 

Class  B-C  Mishaps  1973-1998. 

Results  and  Analysis.  The  beta  coefficients  and  then- 
corresponding  p-values  for  both  the  full  and  backward  stepwise  regression  results  are 
presented  in  Tables  22  and  23.  The  backward  stepwise  regression  analysis  results 
produced  an  R2  =  .8801 .  It  also  showed  that  the  FY*FR  term  was  the  only  term  that  had 
no  significant  effect  on  the  expected  mishap  rate.  The  effect  of  the  flight-related  confound 


Table  22.  Full  Model:  Class  B-C,  1973-1998 


Terms 

Beta 

Beta  Coefficients 

P-Values 

Intercept 

Po 

21.511 

.8041 

FY 

Pi 

-.187 

.8537 

Pi 

P2 

-56.101 

.5309 

P2 

Pa 

43.344 

.6245 

FY*RM 

Pl2 

.627 

.5472 

FY*FR 

Pl3 

.435 

.6756 

the  1973-1983  period  is  the  same  as  that  for  the  1984-1987  period  (FR  =  RM  =  0). 
However,  the  intercepts  are  different.  This  is  consistent  with  the  earlier  results  that 
showed  a  significant  jump  in  the  regression  line  at  1983.  When  FR  =  0  and  RM  =  1,  the 


59 


Table  23.  Stepwise  Results:  Class  B-C,  1973-1998 


Terms 

Beta 

Beta  Coefficients 

P-Values 

Intercept 

1  Po 

57.049 

.0039 

FY 

Pi 

-.602 

.0079 

RM 

P2 

-91.639 

.0025 

FR 

P3 

6.332 

.0048 

FY*RM 

Pl2 

1.043 

.0021 

positive  slope  of  the  model  indicates  that  during  the  years  RM  has  been  in  effect,  the 
Class  B-C  category  mishap  rate  has  increased.  A  discussion  of  the  possibilities  for  this 
increase  was  previously  presented  under  the  piecewise  regression  analysis  for  this  period. 

Class  B-C  Mishaps  1984-1998. 

Results  and  Analysis.  The  beta  coefficients  and  their 
corresponding  p-values  for  both  the  full  model  and  backward  stepwise  regression  results 
are  presented  in  Tables  24  and  25. 


Table  24.  Full  Model:  Class  B-C,  1984-1998 


Terms 

Beta 

Beta  Coefficients 

P- Values 

Intercept 

Po 

21.511 

.5065 

FY 

Pi 

-.187 

.6196 

P2 

-56.101 

.1109 

FY*RM 

Pl2 

.627 

.1241 

Table  25.  Stepwise  Results:  Class  B-C,  1984-1998 


Terms 

Beta 

Beta  Coefficients 

P-Values 

Intercept 

Po 

-29.702 

.0017 

FY 

Pi 

.412 

.0005 

RM 

P2 

-2.245 

.0204 

The  backward  stepwise  regression  analysis  results  produced  an  R2  =  .713 1 .  It  also 
showed  that  the  FY*RM  term  did  not  significantly  contribute  to  predicting  the  expected 
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mishap  rate.  This  model  is  virtually  the  same  as  the  model  produced  in  the  1973-1998 
test  for  this  period.  Again,  the  important  note  of  interest  is  the  positive  slope  indicating 
an  increasing  Class  B-C  mishap  trend  since  1988. 

Summary.  Stepwise  regression  was  employed  to  determine  the  relative 
significance  of  the  variables  for  predicting  the  mishap  rate.  The  results  from  all  tests 
were  consistent  with  those  found  in  previous  analyses.  For  the  Class  A  mishap  category, 
the  rate  is  dependent  only  upon  the  fiscal  year.  For  the  Class  B-C  category  the  mishap 
rate  depends  not  only  upon  the  fiscal  year  but  what  policies  were  in  effect.  Prior  to  1983, 
the  flight-related  confound  was  a  significant  factor.  After  1987,  the  presence  of  RM  was 
a  significant  variable  in  predicting  the  mishap  rate.  Since  a  purpose  of  RM  is  to  prevent 
mishaps,  it  is  unlikely  that  RM  would  contribute  to  a  rising  mishap  rate. 

Since  RM  is  designed  to  prevent  human  error  mishaps,  mishap  causal  data  was 
obtained  for  analysis  of  the  human  error-related  mishaps. 

Analysis  of  Human  Error-Related  Mishaps.  It  is  estimated  that  70  percent  of 
aviation  mishaps  have  human  error  as  at  least  a  causal  factor  (Department  of  Defense, 
1997:31).  Considering  that  one  of  the  purposes  of  RM  is  to  prevent  human  error 
mishaps,  one  might  expect  to  see  a  reduction  in  the  proportion  of  human  error-  related 
mishaps  if  RM  was  effective.  The  Army  causal  data  before  the  implementation  of  RM 
was  not  available.  Therefore,  the  data  from  1990  to  1998  was  analyzed  to  determine  the 
proportion  of  Class  A,  B,  and  C  mishaps  where  human  error  was  at  least  a  cause  factor. 

Results  and  Analysis.  The  results  shown  in  Table  26  are  the  average 
proportions  of  causes  by  mishap  class. 
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Table  26.  Average  Proportion  of  Mishaps  by  Cause  and  Class 


Class 

Human  Error 

Material  Failure 

Environmental 

Undet. 

A 

.770 

.251 

.059 

.070 

B 

.745 

.306 

.102 

.082 

C 

.642 

.225 

.094 

.133 

Note:  Proportions  by  class  do  not  add  to  10C 

percent  because  t 

here  can 

be  more  than  one  cause  for  a  mishap 


Although  the  number  and  rate  of  Class  A  mishaps  has  decreased  in  absolute  terms  over 
the  years,  the  average  proportion  of  human  error-related  Class  A  mishaps  is  comparable 
to  the  average  of  70  percent  cited  above.  The  numbers  and  causes  of  Class  B  and  C 
mishaps  before  1990  were  not  available,  but  the  average  proportion  since  1990  is  also 
comparable  to  70  percent.  Of  note  is  that  as  the  average  proportion  of  human  error 
mishaps  decreases  with  class,  the  proportion  of  undetermined  causes  increases. 

Figures  13  and  14  presents  the  human  error-related  mishap  proportions  from  1990 
to  1998.  As  Figure  13  shows,  the  trend  of  the  proportion  of  human  error-related  mishaps 
is  slightly  increasing.  Earlier  analysis  showed  that  the  overall  Class  A  mishap  rate  was 
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Figure  13.  Class  A  Human  Error-Related  Mishap  Proportions 
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decreasing  for  this  period.  As  Figure  14  shows.  Classes  B  and  C  have  decreasing  trends 
of  human  error-related  mishap  proportions.  Earlier  analysis  showed  that  the  overall  Class 
B-C  mishap  rate  was  increasing  during  this  period.  Without  data  before  1990,  it  is  not 
possible  to  determine  any  relationship  between  a  change  in  the  proportion  of  human 
error-related  mishaps  with  the  presence  of  RM.  The  graph  indicates  that  the  Class  A 
mishaps  related  to  human  error  are  on  a  slight  downward  trend.  This  is  consistent  with 
previous  analysis  that  found  that  the  Class  A  mishap  rate  has  been  steadily  declining 
since  1973.  This  analysis  indicates  that,  since  1990,  as  the  rate  of  human-error  related 
mishaps  declines,  the  overall  Class  A  mishap  rate  declines.  Class  B  mishap  rate  is 
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Figure  14.  Classes  B  and  C  Human  Error-Related  Mishap  Proportions 


constant  and  the  Class  C  mishap  rate  is  on  an  upward  trend.  This  indicates  that  the  Class 
C  human  error-related  mishap  rate  drives  the  upward 

Figure  15  compares  human  error-related  mishap  rate  trends  for  each  category. 
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figure  id.  Human  trror  Mishap  Kate 


trend  in  the  overall  Class  B-C  mishap  rate.  Table  27  summarizes  the  class-human  error 
trends  for  the  1990-1998  period.  If  RM  was  effective,  one  would  expect  not  only 


Table  27.  Class  -  Human  Error  Trends 


Class 

Human  Error  Rate 

Human  Error  Proportion 

A 

Decreasing 

Increasing 

B 

Constant 

Decreasing 

C 

Increasing 

Decreasing 

the  human  error  rate  to  decrease  but  the  human  error  proportion  to  decrease  as  well. 

With  the  data  available,  it  is  only  possible  to  see  what  occurred  after  RM  took  effect. 

Summary.  This  section  analyzed  the  causal  data  from  1990-1998  to  determine 
trends  in  the  rates  and  proportions  of  human  error-related  mishaps.  A  summary  of  trends 
was  presented  in  Table  16.  If  RM  is  to  have  any  effect  on  the  mishap  rate,  then  RM 
should  reduce  the  human  error-related  mishap  rate.  Additionally,  the  proportion  of 
human  error-related  mishaps  should  decline.  No  specific  conclusions  concerning  the 
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effect  of  RM  on  the  human  error-related  causes  could  be  drawn  without  causal  data 
before  1987. 

Summary 

This  chapter  is  the  focal  point  of  the  research  project.  It  addressed  the  sources 
and  description  of  the  data.  It  then  pointed  out  various  potential  threats  to  the  internal 
and  external  validity  of  the  research.  The  largest  threat  to  the  overall  validity  of  the 
research  is  history.  Specifically,  any  reduction  in  the  mishap  rate  must  first  be  attributed 
to  the  mishap  prevention  factors  described  in  Chapter  II:  leadership,  mishap 
investigation,  technological  advances,  and  cockpit  resource  management.  Any  remaining 
contribution  can  be  attributed  to  the  impact  of  RM.  Last,  it  presented  the  methodology, 
results,  and  analysis  of  the  data.  Four  types  of  statistical  tests  were  performed  to  analyze 
the  data:  1 )  comparison  of  means  of  percentage  period  indexes  2)  comparison  of  the 
residual  variances  of  the  mishap  rate  values,  3)  discontinuous  piecewise  linear  regression, 
and  4)  stepwise  regression.  Finally,  an  analysis  of  the  proportions  and  rates  of  human 
error-related  mishaps  by  cause  type  was  conducted. 

The  comparison  of  means  test  found  that  the  mean  PPIs  for  each  time  period  for 
the  Class  A  mishap  category  were  not  significantly  different.  The  same  was  true  for  the 
Class  B-C  mishap  category.  The  residual  variance  test  found  that  for  the  Class  A  mishap 
category,  there  was  no  significant  difference  in  the  mishap  rate  process.  However,  for  the 
Class  B-C  category  the  residual  variance  for  the  1988-1998  time  period  was  significantly 
less  than  that  of  the  1973-1987  time  period.  This  indicated  the  occurrence  of  a  process 
change.  The  piecewise  linear  regression  analysis  results  indicated  that  the  change  in  the 
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mishap  rate  process  could  not  be  attributed  to  the  presence  of  RM.  Stepwise  regression 
analysis  confirmed  the  piecewise  analysis  results.  The  human  error-related  mishap 
proportions  analysis  found  was  an  upward  trend  since  1990  in  the  Class  A  category. 
However,  for  the  Class  B  and  C  categories,  the  proportions  have  declined.  The  Class  A 
human  error-related  mishap  rate  has  declined  since  1990.  This  is  consistent  with  the  fact 
that  the  overall  Class  A  mishap  rate  has  been  on  a  downward  trend  since  1973.  In  the 
Class  B  mishap  category,  the  rate  has  been  constant.  In  the  Class  C  mishap  category, 
human  error-related  mishaps  have  driven  the  upward  Class  B-C  mishap  rates  since  1990. 

Based  on  the  results,  analysis,  and  answers  to  the  three  investigative  questions 
addressed  in  this  chapter,  overall  research  conclusions  and  recommendations  can  be 
made.  These  are  presented  in  Chapter  IV. 
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IV.  Conclusions  And  Recommendations 


Overview 

The  Air  Force  Class  A  aviation  mishap  rate  has  been  steadily  holding  at  about  1 .5 
mishaps  per  100,000  flight  hours  since  the  mid-1980's.  Due  to  its  increasingly  limited 
resources  and  recent  increased  concern  over  flight  safety,  the  Air  Force  is  seeking  ways 
of  reducing  aviation  mishaps.  The  Army's  Class  A  mishap  rate,  historically  higher  than 
that  of  the  Air  Force  (Appendix  B),  has  declined  to  the  point  where  it  dipped  below  that 
of  the  Air  Force.  Since  the  Army  introduced  RM  in  1987,  it  has  considered  RM  as  a 
contributing  factor  to  the  Class  A  mishap  rate  reduction.  The  Air  Force  completed  initial 
implementation  of  ORM  in  1998  and  is  expecting  to  see  its  Class  A  mishap  rate  decrease. 
This  research  sought  to  determine  what  expectations  the  Air  Force  can  have  regarding  the 
effect  of  ORM  on  its  aviation  mishap  rate  by  examining  the  Army’s  implementation  of 
RM  and  its  aviation  mishap  data.  This  chapter  first  reviews  findings  based  on  the 
answers  to  the  investigative  questions.  It  then  addresses  the  original  research  question 
and  presents  conclusions  based  on  the  overall  research.  Finally,  it  makes 
recommendations  for  future  research. 

Findings 

Three  findings  are  presented  based  on  the  research  to  answer  the  five  investigative 
questions  posed  in  Chapter  I. 

Finding  1.  The  first  research  question  sought  to  determine  the  major  factors  that 
influence  military  aviation  safety.  Addressed  in  Chapter  II,  these  factors  were  divided 
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into  two  types:  mishap  cause  factors  and  mishap  prevention  factors.  The  mishap  cause 
factors  were  based  on  the  DoD  mishap  categories:  human  factor  (error),  material  failure, 
environmental,  and  other.  A  fifth  factor,  though  not  significantly  substantiated,  was 
operations  tempo.  The  mishap  prevention  factors  were  narrowed  down  to  just  four: 
leadership,  mishap  investigation,  advancement  in  technology,  and  human  error 
prevention  programs. 

Finding  2.  The  second  investigative  question  sought  to  compare  the  Army's 
implementation  of  RM  with  the  Air  Force's  implementation  of  ORM.  Also  addressed  in 
Chapter  II,  the  comparison  was  based  on  three  criteria:  published  directives, 
responsibility,  and  training.  The  literature  review  found  two  minor  differences  in  how 
the  Army  and  Air  Force  have  implemented  risk  management.  First,  there  was  a  four  year 
delay  in  the  Army's  integration  of  RM  training.  The  Air  Force  implemented  ORM 
training  from  the  start.  Second,  the  Army's  initial  focus  of  RM  application  was  to  the 
acquisition,  aviation,  and  field  training  areas.  The  Air  Force  ORM  implementation  began 
in  all  operations  and  processes.  Despite  these  differences,  the  Army  and  Air  Force 
implementations  have  not  been  significantly  different  from  each  other. 

Finding  3.  Chapter  III  addressed  the  last  three  investigative  questions  using 
statistical  analysis.  The  first  of  these  asked  whether  there  was  any  significant  difference 
in  the  mishap  rate  after  die  implementation  of  RM.  In  absolute  terms,  the  rates  are  lower, 
but  the  comparison  of  means  of  the  period  percentage  indexes  and  the  comparison  of 
residual  variances  tests,  showed  that  for  the  Class  A  mishap  rate,  there  was  no  significant 
difference  in  the  process  over  the  years.  Analysis  of  the  Class  B-C  mishap  rate,  however, 
showed  that  there  was  a  significant  difference  in  the  process. 
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Finding  3a.  Since  there  was  a  difference  in  the  Class  B-C  mishap  category,  the 
analysis  then  focused  on  whether  this  change  was  due  to  RM.  Although  there  was  no 
difference  in  the  Class  A  mishap  category,  the  analysis  was  performed  to  confirm  the 
previous  results.  The  Class  A  mishap  rate  had  been  on  a  downward  trend  before  1 987 
and,  as  the  discontinuous  piecewise  linear  regression  analysis  confirmed,  there  was  no 
change  in  the  trend  after  1987.  The  Class  B-C  mishap  rate  had  also  been  on  a  downward 
trend  before  1987  but  actually  began  to  rise  since.  The  analysis  thus  found  that  this 
change  was  not  due  to  RM. 

Finding  3h.  The  flight-related  confound  was  not  expected  at  the  outset  of  this 
research  since  it  was  discovered  during  data  collection.  Additional  analysis  had  to  be 
performed  to  determine  its  effect  on  the  mishap  rate  and,  in  turn,  any  effect  the  confound 
had  on  masking  any  effect  RM  may  have  had  on  the  mishap  rate.  The  analysis  found  that 
for  the  Class  A  mishap  category,  the  confound  had  no  effect  on  the  rate.  In  turn,  it  did 
not  affect  any  impact  RM  may  have  had  on  the  mishap  rate.  However,  for  the  Class  B-C 
mishap  category,  the  confound  did  have  a  significant  effect  on  the  mishap  rate. 
Nevertheless,  this  effect  did  not  appear  to  mask  any  impact  RM  had  on  the  mishap  rate. 
The  confound  may  partially  explain  the  apparent  Class  B-C  mishap  rate  increase  since 
1987.  Overall,  it  does  not  appear  that  RM  has  had  much,  if  any  effect  on  the  Army 
aviation  mishap  rate. 

Conclusions 

The  purpose  of  this  research  was  to  determine,  based  on  the  Army's  experience 
with  RM,  whether  the  Air  Force  should  expect  a  reduction  in  its  aviation  mishap  rate 
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after  the  implementation  of  ORM.  Based  on  the  findings  from  the  research  conducted, 
the  Air  Force  should  not  expect  to  see  a  significant  decline  in  its  mishap  rate  due  to  the 
implementation  of  ORM. 

There  are  two  possible  reasons  the  effect  of  RM  does  not  appear  in  the  mi  chap 
rates.  The  first  may  be  attributed  to  the  findings  from  Chapter  II.  The  Army  began 
implementation  of  RM  in  1987  but  there  was  a  delay  of  four  years  before  RM  was 
integrated  into  training  and  doctrine.  Furthermore,  responsibility  for  full  integration  was 
not  formally  assigned  until  1997.  It  may  be  several  more  years  before  the  effect  of  RM  is 
reflected  in  its  aviation  mishap  rates.  The  second  reason  is  the  effect  of  history.  As 
described  earlier  in  this  chapter,  many  events  have  occurred  contemporaneous  with  RM 
and  since  its  inception  that  may  also  have  affected  the  mishap  rate.  It  is  difficult  to 
differentiate  the  effect  of  RM  from  that  of  aircrew  coordination  training  (ACT)  since  both 
are  aimed  at  reducing  human  error  mishaps.  However,  the  fact  that  ACT  was  not 
officially  implemented  until  1994  strengthens  the  argument  that  the  effect  of  RM  has  not 
been  reflected  in  the  mishap  rate.  As  a  whole  then,  the  mishap  prevention  factors 
previously  described  were  reducing  the  mishap  rate  before  and  since  1987.  The 
combination  of  these  two  reasons  resulted  in  the  undetectable  effect  of  RM  on  the 
aviation  mishap  rate. 

Recommendations 

Based  on  the  preceding  research,  two  recommendations  are  made. 

Recommendation  1.  The  Army  should  continue  with  the  integration  and 
application  of  RM  throughout  its  operations.  The  groundwork  has  been  laid  in  its 
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doctrine,  education  and  training  system,  acquisitions  process,  and,  specifically,  its 
aviation  processes.  Much  is  yet  to  be  done.  Although  the  long-term  mishap  statistics  do 
not  appear  to  show  it  yet,  an  effective  RM  program  (all  else  being  equal)  should  yield  a 
noticeable  difference  in  the  future.  If  the  rate  of  human  error-related  mishaps  declines, 
the  Class  C  mishap  rate  should  reverse  course  and  start  a  downward  trend. 

Recommendation  2.  The  second  recommendation  is  for  the  Air  Force  to  leam  as 
much  about  RM  as  it  can  from  the  Army  (and  other  sister  services).  The  Air  Force 
should  not  expect  an  immediate  reduction  in  its  Class  A  mishap  rate  due  to  ORM.  In 
addition,  the  Air  Force  may  not  see  any  improvements  in  its  mishap  rate  at  all. 

Future  Research 

The  Army  and  Air  Force  could  both  benefit  from  future  research  relating  the 
effect  of  risk  management  to  the  aviation  mishap  rate. 

Benefit  1.  The  Army  data  required  to  perform  a  more  in-depth  analysis  was  not 
available  to  the  researcher.  It  would  be  worthwhile  to  look  at  the  causal  data  before  1987 
and  compare  it  with  the  causal  data  after  1987.  If  the  rates  and  proportions  of  human 
error  mishaps  declined,  it  may  be  that  RM  had  a  more  significant  impact  than  what  this 
research  showed. 

Benefit  2.  It  may  also  be  beneficial  to  repeat  this  analysis  using  a  break  point 
other  than  1987.  Since  much  of  the  training  and  integration  occurred  several  years  after 
initial  implementation,  a  more  appropriate  break  point  may  be  defined.  Results  may 
show  that  after  full  integration  of  RM,  and  a  statistically  significant  number  of  years  after 
the  new  break  point,  the  effect  of  RM  would  be  reflected  in  the  mishap  rate. 
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Benefit  3.  Since  one  of  the  Air  Force  leaderships'  motivations  for  implementing 
ORM  was  to  reduce  the  mishap  rate,  an  analysis  on  the  Air  Force  aviation  mishap  rate 
similar  to  that  performed  in  this  study  could  be  useful.  Similarly,  since  the  Navy  and 
Marines  are  also  employing  ORM,  it  would  be  useful  to  see  what  effect  it  has  had  on 
their  aviation  mishap  rates. 

Summary 

This  chapter  presented  the  research  findings,  conclusions,  recommendations,  and 
future  research  possibilities.  This  research  focused  on  the  impact  of  RM  on  the  Army 
aviation  mishap  rate  to  determine  what  the  Air  Force  should  expect  from  ORM  in  terms 
of  a  reduced  aviation  mishap  rate.  It  concluded  that  the  Air  Force  should  not  expect  a 
significant  reduction  in  its  aviation  mishap  rate  due  to  the  implementation  of  ORM. 
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Appendix  A:  Army  Data 
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Appendix  B:  Air  Force  Data 
Table  B-l.  Air  Force  Flight  Mishap  Data 
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Table  B-2.  Average  Proportion  of  Mishap  Causes  by  Class,  1993-1998 
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Figure  B-l.  USAF  Mishap  Reason  Proportions 


Figure  B-2.  Army  and  Air  Force  Class  A  Mishap  Rates 
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Appendix  C:  Class  A  Relative  Frequency  Distribution 
Including  1991 


Residuals  Class  A  Rate 


Moments 


Mean 

0.00000 

Std  Dev 

0.49067 

Std  Error  Mean 

0.09623 

Upper  95%  Mean 

0.19818 

Lower  95%  Mean 

-0.19818 

N 

26.00000 

Sum  Weights 

26.00000 

Test  for  Normality 

Shapiro- Wilk  W  Test 

W  Prob<W 
0.803529  0.0001 
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Appendix  D:  Class  A  Relative  Frequency  Distribution 
Excluding  1991 


Residuals  Class  A  73-98  (-91) 


Moments 


Mean 

-0.00000 

StdDev 

0.31333 

Std  Error  Mean 

0.06267 

Upper  95%  Mean 

0.12933 

Lower  95%  Mean 

-0.12933 

N 

25.00000 

Sum  Weights 

25.00000 

Test  for  Normality 

Shapiro- Wilk  W  Test 

W 

Prob<W 

0.966640 

0.5714 
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Appendix  E:  Class  B-C  Relative  Frequency  Distribution 


Residuals  Class  B-C  Rate 


Moments 


Mean 

0.00000 

Std  Dev 

3.72033 

Std  Error  Mean 

0.74407 

Upper  95%  Mean 

1.53566 

Lower  95%  Mean 

-1.53566 

N 

25.00000 

Sum  Weights 

25.00000 

Test  for  Normality 


Shapiro- Wilk  W  Test 

W  Prob<W 

0.943593  0.1882 
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Appendix  F:  Comparison  of  Means  Percentage  Period  Index  Test  Results 
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Diff  means 


t-test  1.601 


t-Crit  1.714 


means  are  equal 
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Appendix  G:  Comparison  of  Variance  Test  Results 


Class  A _  Class  B-C 
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Appendix  H:  Stepwise  Regression  Analysis  Results 
The  full  mathematical  model  for  the  1973-1998  time  period  for  both  Class  A  and  Class 
B-C  categories  was: 

E(MR)  =  p0  +  pi*FY  +  p2*RM  +  p3*FR  +  p12*FY*RM  +  Pi3*FY*FR 
The  full  mathematical  model  for  1984-1998  time  period  for  both  Class  A  and  Class  B-C 
categories  was: 

E(MR)  =  p0  +  Pi*FY  +  p2*RM  +  p3*FY*RM 
Class  A 

1973-1998:  E(MR)=  11.19  -  .104*FY 

1984-1998  E(MR)=  1 1.826 -.111*FY 

Class  B-C  1973-1998 

E(MR)  =  57.049  -  .602*FY  -  91.639*RM  +  6.332*FR  +  1.043*FY*RM 
When  FR  =  1  and  RM  =  0  (1973-1983): 

E(MR)  =  63.381  -  .602*FY 
When  FR  =  0  and  RM  =  0  (1984-1987): 

E(MR)  =  57.049  -.602*FY 
When  FR  =  0  and  RM  =  1  (1988-1998): 

E(MR)  = -34.59 +  .441*FY 
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1984-1998  Class  B-C 


E(MR)  =  -29.702  +  .412*FY  +  -2.245*RM 
When  RM  =  0(1984-1987): 

E(MR)  =  -29.702  +  ,412*FY 

WhenRM=l  (1988-1998): 

E(MR)  =  -31.94  +  .412*FY 
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